Space-Ground Integrated Cache Update and Collaborative Transmission Method, Device and Storage Medium

Abstract

This invention discloses a satellite-ground integrated cache update and cooperative transmission method and related apparatus. The method includes: establishing three cooperative transmission links: a base station-user link, a satellite-base station link, and a gateway-satellite shared link; and establishing a channel model based on a probability density function. Based on the channel model, an optimization problem is established with the cached content of the satellite and base station layers as optimization variables, aiming to maximize the long-term system energy efficiency. The optimization problem is solved using a deep deterministic policy gradient algorithm, and behavioral cloning technology is used to improve the training performance of the deep deterministic policy gradient algorithm, ultimately obtaining the optimal satellite-ground integrated cooperative caching strategy. This invention effectively improves the energy efficiency of the satellite-ground integrated network system.

Description

Technical Field

[0001] This invention relates to the field of wireless communication technology, and in particular to a satellite-ground integrated cache update and cooperative transmission method, apparatus and storage medium. Background Technology

[0002] As a future mainstream network architecture, integrated space-ground caching networks can effectively alleviate network traffic pressure and reduce system overhead through their edge caching devices. However, due to limited edge caching space, integrated space-ground networks face the challenge of low energy efficiency. Inappropriate caching strategies not only reduce the utilization efficiency of system resources but also lead to further degradation in caching efficiency over long-term operation due to dynamic changes in user demand and channel parameters, thus affecting the overall system performance. Summary of the Invention

[0003] This invention provides a satellite-ground integrated cache update and collaborative transmission method and related apparatus, which aims to effectively improve the energy efficiency of the system.

[0004] This invention provides a satellite-ground integrated cache update and cooperative transmission method, the method comprising the following steps:

[0005] Construct a space-ground integrated caching network system, and establish three cooperative transmission links based on the space-ground integrated caching network system: base station-user link, satellite-base station link, and gateway-satellite shared link. Establish a channel model for these three cooperative transmission links based on the probability density function.

[0006] Based on the channel model, an optimization problem P1 is established with the cache contents of satellite and base station layers as optimization variables and the goal of maximizing long-term system energy efficiency.

[0007] The optimization problem P1 is solved using the Deep Deterministic Policy Gradient Algorithm (DPRQA), and behavioral cloning techniques are employed to improve the training performance of DPRQA. During the solution process, based on the asynchronous update mechanism of satellites and base stations, optimization problem P1 is decomposed into two sub-problems: base station cache content update P1.1 and satellite cache content update P1.2. Sub-problem P1.1 uses the DPRQA to obtain the cache decision strategy under the current base station cache content distribution, based on the predicted user request information within the base station update cycle. Sub-problem P1.2 uses the DPRQA to obtain the satellite cache decision strategy based on the current base station cache strategy and the predicted user request information within the update cycle.

[0008] Through iterative training, the deep deterministic strategy gradient algorithm for updating base station cache content subproblem P1.1 and satellite cache content subproblem P1.2 converges, ultimately obtaining the optimal satellite-ground integrated cooperative caching strategy.

[0009] Optionally, the specific implementation steps of the method include:

[0010] Step 1: Based on the constructed satellite-ground integrated caching network system, calculate the system energy efficiency expression using channel parameters;

[0011] Step 2: Initialize the network parameters for the deep deterministic policy gradient algorithm at the satellite and base station layers;

[0012] Step 3: Collect a segment of historical user request information, select an expert strategy to generate the corresponding base station layer expert caching scheme and satellite expert caching scheme, assemble them into {user request information, expert caching scheme}, and store them in the base station layer expert experience cache pool and the satellite expert experience cache pool respectively.

[0013] Step 4: Randomly select a set of samples from the base station layer expert experience cache pool and the satellite expert experience cache pool respectively, and pre-train the Actor network in the deep deterministic policy gradient algorithm using the backpropagation mechanism in supervised learning;

[0014] Step 5: Determine if the number of training rounds exceeds the set number of expert training rounds. If not, repeat step 3. If the set number of expert training rounds is reached, the Actor network is considered to be pre-trained and the parameters of the Actor network are copied to the target Actor network.

[0015] Step 6: Determine whether the current stage is satellite update or base station update. If it is satellite update, proceed to step 7; if it is base station update, proceed to step 11.

[0016] Step 7: Collect the predicted values ​​of user request information in the current satellite phase, as well as the network parameters of the current satellite depth deterministic strategy gradient algorithm, satellite experience cache pool, and base station cache decision strategy.

[0017] Step 8: Generate a base station caching scheme based on the current base station caching decision strategy and the predicted value of user request information, and combine {current satellite caching scheme, base station caching scheme, user request information} into the input state of the satellite depth deterministic policy gradient algorithm, obtain the output action, and observe the long-term network system energy efficiency as a reward after taking the action;

[0018] Step 9: Put the experience samples {state, action, reward, next state} into the satellite experience cache pool, and randomly select a set of samples from it to train the deep deterministic policy gradient algorithm, and update the network parameters of the deep deterministic policy gradient algorithm.

[0019] Step 10: Determine if the training count has reached the set maximum number of times. If not, repeat step 8. If the set maximum number of times has been reached, the current satellite depth deterministic policy gradient algorithm is considered to have been trained, the current satellite cache decision content is output, and the process jumps to step 15.

[0020] Step 11: Collect the predicted values ​​of user request information in the current base station phase, as well as the network parameters of the current base station deep deterministic strategy gradient algorithm, the base station experience cache pool, and the satellite caching scheme;

[0021] Step 12: Combine {current satellite caching scheme, base station caching scheme, user request information} into the input state of the base station deep deterministic policy gradient algorithm, obtain the output action, and observe the short-term network system energy efficiency as a reward after taking the action.

[0022] Step 13: Put the experience samples {state, action, reward, next state} into the base station experience cache pool, and randomly select a set of samples from it to train the deep deterministic policy gradient algorithm, and update the network parameters of the deep deterministic policy gradient algorithm.

[0023] Step 14: Determine whether the training count has reached the set maximum number of times. If not, repeat step 11. If the set maximum number of times has been reached, it is determined that the training of the current base station deep deterministic policy gradient algorithm is complete. Output the current base station cache decision content and jump to step 15.

[0024] Step 15: Determine if there is no new update phase. If there is a new update phase, repeat step 6; otherwise, end.

[0025] Optionally, the optimization problem P1, which aims to maximize the long-term system energy efficiency, is specifically expressed as:

[0026]

[0027] Where T is the index of the satellite update cycle, and let the satellite update cycle T contain W base station update cycles t, that is... This represents the cache matrix of base stations within the satellite update period T and the base station update period t. This represents the cache vector of satellites within the base station update period t within the satellite update period T. This represents the total amount of user request data in the network. This represents the total energy consumption of the system. C1 represents the base station's buffer space limit. C1 represents the maximum buffer space of the base station, and C2 represents the buffer space limit of the satellite. C3 indicates the maximum cache space of the satellite, and C3 indicates the maximum amount of cached data for a file on any cache device that is sufficient to recover the original file.

[0028] Optionally, the expressions for the base station cache content update subproblem P1.1 and the satellite cache content update subproblem P1.2 are as follows:

[0029]

[0030] stC1,C2,C3

[0031]

[0032] stC1,C2,C3.

[0033] Optionally, the solutions to the base station cache content update subproblem P1.1 and the satellite cache content update subproblem P1.2 are iterated by repeating the following steps until the algorithm gradually converges:

[0034] fixed Decision-making strategy, solve Make maximum;

[0035] fixed Decision-making strategy, solve Make maximum.

[0036] Optionally, a deep deterministic policy gradient algorithm is used to obtain the base station caching decision strategy under the current satellite cache content distribution, including:

[0037] Base station status: Consists of the current satellite cache decision content and the predicted value of user request information, i.e.: in Let be the predicted probability of file request for file i in cell n during satellite period T and base station period t. The cache vector of satellites within the base station update period t, which is within the satellite update period T. The cache matrix of the base station within the previous update cycle t-1 within the satellite update cycle T;

[0038] Base station actions: Actions are represented by the proportion of cached data for each file stored in each base station, i.e.: in

[0039] Base station incentive: The incentive is represented by the system's short-term energy efficiency, i.e.: This represents the total amount of user request data in the network. This indicates the total energy consumption of the system.

[0040] Optionally, the satellite caching decision strategy is obtained using a deep deterministic policy gradient algorithm, including:

[0041] Satellite status: in Represents state-based As input, the base station cache update decision for satellite update slots is obtained using the output of the deep deterministic policy gradient algorithm. This represents the cache vector of the base station in the last base station update cycle within satellite cycle T-1. This represents the cache vector of satellites within the base station update period t within the satellite update period T-1, where in

[0042] Satellite actions: Actions are represented by the proportion of cached data for each file stored in the satellite, i.e.:

[0043] Satellite rewards: Rewards are represented by the system's long-term energy efficiency, i.e.: This represents the total amount of user request data in the network. This indicates the total energy consumption of the system.

[0044] A second aspect of the present invention provides a satellite-ground integrated cache update and cooperative transmission device, the device comprising:

[0045] The network system construction module is used to construct a space-ground integrated caching network system. Based on the space-ground integrated caching network system, three cooperative transmission links are established: a base station-user link, a satellite-base station link, and a gateway-satellite shared link. A channel model is established for these three cooperative transmission links based on a probability density function.

[0046] The objective optimization problem construction module is used to establish an optimization problem P1 based on the channel model, with the cache contents of the satellite and base station layers as optimization variables and the objective of maximizing the long-term system energy efficiency.

[0047] The target optimization problem solving module is used to solve the optimization problem P1 based on the deep deterministic policy gradient algorithm. Behavioral cloning technology is employed to improve the training performance of the deep deterministic policy gradient algorithm. During the solution process, based on the asynchronous update mechanism of satellites and base stations, the optimization problem P1 is decomposed into two sub-problems: base station cache content update P1.1 and satellite cache content update P1.2. Sub-problem P1.1 uses the deep deterministic policy gradient algorithm to obtain the cache decision strategy under the current base station cache content distribution, based on the predicted user request information within the base station update cycle. Sub-problem P1.2 uses the deep deterministic policy gradient algorithm to obtain the satellite cache decision strategy based on the current base station cache strategy and the predicted user request information within the update cycle.

[0048] The caching strategy acquisition module is used to converge the deep deterministic strategy gradient algorithm for the base station cache content update subproblem P1.1 and the satellite cache content update subproblem P1.2 through iterative training, and finally obtain the optimal satellite-ground integrated collaborative caching strategy.

[0049] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the space-ground integrated cache update and cooperative transmission method as described in any one of the first aspects of the present invention.

[0050] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the space-ground integrated cache update and cooperative transmission method as described in any of the first aspects of the present invention.

[0051] As can be seen from the above technical solutions, the present invention has the following advantages:

[0052] This invention employs encoding caching technology to process cached files, improving caching efficiency. It models the channel model based on probability density functions, fully considering the dynamic changes of the channel model during long-term network operation. It uses behavioral cloning technology to pre-train the Deep Deterministic Policy Gradient Algorithm (DDPG), solving the reward sparsity problem faced by the DDPG algorithm in satellite-ground integrated network optimization. Furthermore, it uses a dual DDPG algorithm to make caching decisions for the cached content of the base station layer and satellites, combining them with an iterative approach to address the asynchronous update problem of base stations and satellites, a topic rarely addressed in current research on satellite-ground integrated networks using deep reinforcement learning. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating the implementation steps of a satellite-ground integrated cache update and cooperative transmission method provided in this embodiment of the invention;

[0055] Figure 2 This is a communication topology diagram of the integrated space-ground caching network provided in an embodiment of the present invention;

[0056] Figure 3 This is a structural block diagram of the space-ground integrated cache update and cooperative transmission device provided in an embodiment of the present invention;

[0057] Figure 4 A performance comparison chart of the caching decision algorithm in a small-scale ISTN is provided for the verification embodiments of this invention;

[0058] Figure 5 The energy efficiency performance comparison chart of various caching decision algorithms under different file numbers provided in the verification embodiments of this invention is shown.

[0059] Figure 6 This is a comparison chart of the energy efficiency performance of various caching decision algorithms under different numbers of base stations, provided as a verification embodiment of the present invention. Detailed Implementation

[0060] This invention provides a method and related apparatus for integrated satellite-ground cache update and cooperative transmission, aiming to solve the optimization problem of two-level cache devices at the base station layer and satellite layer in a dynamic environment integrated satellite-ground network.

[0061] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0062] As a future mainstream network architecture, integrated space-ground caching networks can effectively alleviate network traffic pressure and reduce system overhead through their edge caching devices. However, due to limited edge caching space, integrated space-ground networks face the challenge of low energy efficiency. Inappropriate caching strategies not only reduce the utilization efficiency of system resources but also lead to further degradation in caching efficiency over long-term operation due to dynamic changes in user demand and channel parameters, thus affecting the overall system performance.

[0063] To address this complex energy efficiency optimization problem, this invention employs a deep reinforcement learning approach to design a caching decision-making scheme. This scheme dynamically adapts to environmental changes and improves caching efficiency through intelligent methods. Furthermore, to further enhance the utilization of cache resources, the scheme introduces an encoding caching technique based on Luby Transform Code (LT Code) to optimize caching performance and improve the system's resource allocation efficiency under limited cache conditions.

[0064] Integrated space-ground caching networks involve collaborative caching and delivery between two layers of caching devices. Inappropriate distribution of cached content can reduce the offloading performance of limited cache space for network traffic. Existing research largely focuses on static caching designs based on known and fixed content popularity, which is difficult to adapt to dynamic real-world scenarios. Furthermore, current research primarily focuses on complete file transfers, with insufficient attention paid to the application of coded caching techniques. Regarding cache updates, most deep reinforcement learning-based methods only consider the case of synchronous updates between base stations and satellites. To address these issues, this study proposes a dynamic caching strategy based on deep reinforcement learning, supporting asynchronous updates between base stations and satellites to maximize system energy efficiency and improve overall performance.

[0065] The satellite-ground integrated cache update and cooperative transmission method provided in Embodiment 1 of the present invention includes the following steps:

[0066] Construct a space-ground integrated caching network system, and establish three cooperative transmission links based on the space-ground integrated caching network system: base station-user link, satellite-base station link, and gateway-satellite shared link. Establish a channel model for these three cooperative transmission links based on the probability density function.

[0067] Based on the channel model, namely the data transmission volume and power consumption of the three links, an optimization problem P1 is established with the cached content of the satellite and base station layers as optimization variables and the goal of maximizing the long-term system energy efficiency.

[0068] The optimization problem P1 is solved using the Deep Deterministic Policy Gradient Algorithm (DPRQA), and behavioral cloning techniques are employed to improve the training performance of DPRQA. During the solution process, based on the asynchronous update mechanism of satellites and base stations, optimization problem P1 is decomposed into two sub-problems: base station cache content update P1.1 and satellite cache content update P1.2. Sub-problem P1.1 uses the DPRQA to obtain the cache decision strategy under the current base station cache content distribution, based on the predicted user request information within the base station update cycle. Sub-problem P1.2 uses the DPRQA to obtain the satellite cache decision strategy based on the current base station cache strategy and the predicted user request information within the update cycle.

[0069] Through iterative training, the deep deterministic strategy gradient algorithm for updating base station cache content subproblem P1.1 and satellite cache content subproblem P1.2 converges, ultimately obtaining the optimal satellite-ground integrated cooperative caching strategy.

[0070] Specifically, such as Figure 1 As shown, the specific implementation steps of the method include:

[0071] Step 1: Based on the constructed satellite-ground integrated caching network system, calculate the system energy efficiency expression using channel parameters;

[0072] Step 2: Initialize the network parameters for the deep deterministic policy gradient algorithm at the satellite and base station layers;

[0073] Step 3: Collect a segment of historical user request information, select an expert strategy to generate the corresponding base station layer expert caching scheme and satellite expert caching scheme, assemble them into {user request information, expert caching scheme}, and store them in the base station layer expert experience cache pool and the satellite expert experience cache pool respectively.

[0074] Step 4: Randomly select a set of samples from the base station layer expert experience cache pool and the satellite expert experience cache pool respectively, and pre-train the Actor network in the deep deterministic policy gradient algorithm using the backpropagation mechanism in supervised learning;

[0075] Step 5: Determine if the number of training rounds exceeds the set number of expert training rounds. If not, repeat step 3. If the set number of expert training rounds is reached, the Actor network is considered to be pre-trained and the parameters of the Actor network are copied to the target Actor network.

[0076] Step 6: Determine whether the current stage is satellite update or base station update. If it is satellite update, proceed to step 7; if it is base station update, proceed to step 11.

[0077] Step 7: Collect the predicted values ​​of user request information in the current satellite phase, as well as the network parameters of the current satellite depth deterministic strategy gradient algorithm, satellite experience cache pool, and base station cache decision strategy.

[0078] Step 8: Generate a base station caching scheme based on the current base station caching decision strategy and the predicted value of user request information, and combine {current satellite caching scheme, base station caching scheme, user request information} into the input state of the satellite depth deterministic policy gradient algorithm, obtain the output action, and observe the long-term network system energy efficiency as a reward after taking the action;

[0079] Step 9: Put the experience samples {state, action, reward, next state} into the satellite experience cache pool, and randomly select a set of samples from it to train the deep deterministic policy gradient algorithm, and update the network parameters of the deep deterministic policy gradient algorithm.

[0080] Step 10: Determine if the training count has reached the set maximum number of times. If not, repeat step 8. If the set maximum number of times has been reached, the current satellite depth deterministic policy gradient algorithm is considered to have been trained, the current satellite cache decision content is output, and the process jumps to step 15.

[0081] Step 11: Collect the predicted values ​​of user request information in the current base station phase, as well as the network parameters of the current base station deep deterministic strategy gradient algorithm, the base station experience cache pool, and the satellite caching scheme;

[0082] Step 12: Combine {current satellite caching scheme, base station caching scheme, user request information} into the input state of the base station deep deterministic policy gradient algorithm, obtain the output action, and observe the short-term network system energy efficiency as a reward after taking the action.

[0083] Step 13: Put the experience samples {state, action, reward, next state} into the base station experience cache pool, and randomly select a set of samples from it to train the deep deterministic policy gradient algorithm, and update the network parameters of the deep deterministic policy gradient algorithm.

[0084] Step 14: Determine whether the training count has reached the set maximum number of times. If not, repeat step 11. If the set maximum number of times has been reached, it is determined that the training of the current base station deep deterministic policy gradient algorithm is complete. Output the current base station cache decision content and jump to step 15.

[0085] Step 15: Determine if there is no new update phase. If there is a new update phase, repeat step 6; otherwise, end.

[0086] In practice, three cooperative transmission links are introduced into the integrated space-ground caching network to provide file delivery services for users, ensuring that user file requests can be fulfilled. Figure 2 As shown. Then, by deriving formulas for the data transmission volume and power consumption of the three links, an optimization problem is established with the satellite and base station layer cache content as optimization variables and the goal of maximizing long-term system energy efficiency. This problem is solved using the Deep Deterministic Policy Gradient (DDPG) algorithm. To address the reward sparsity problem in effectively improving energy efficiency in complex network systems, behavioral cloning technology is used to improve algorithm training performance. Simultaneously, based on the asynchronous update mechanism of satellites and base stations, the optimization problem is decomposed into base station cache content update and satellite cache content update sub-problems. The base station layer uses the DDPG algorithm to obtain the cache decision strategy under the current base station layer cache content distribution based on the predicted user request information within its update cycle. Similarly, the satellite DDPG algorithm optimizes the satellite cache decision strategy based on the current base station layer cache strategy and the predicted user request information within the update cycle. Finally, through iterative training over multiple update cycles, the two-level DDPG algorithm converges to an efficient satellite-ground integrated collaborative caching strategy.

[0087] Establish a channel model:

[0088] Encoding caching technology is used to cache popular files, allowing files to be cached in the cache device at any ratio. All file requests within the defined scenario belong to a file repository containing F files. The file index is represented by i, and each file has the same size. Let k re To the extent that the amount of data in the source file is sufficient to recover, The cache matrix refers to the base station n base station update matrix within the satellite update cycle T and the base station update cycle t, where Similarly Refers to the cache vector of the satellite within the satellite update cycle T, where

[0089] Base station-user link: In the downlink of the terrestrial network, the effects of both large-scale and small-scale fading are considered. Large-scale fading can be modeled as path loss fading. Small-scale fading can be modeled using a Rayleigh channel. Therefore, the channel gain of the base station delivery channel can be expressed as:

[0090] g n =d -α |h n | 2 (1)

[0091] Where h n The coefficients of the Rayleigh channel are represented by k, which obey the cumulative distribution k of the Rayleigh channel. re Distribution Function (CDF): Therefore, the effective transmission rate of base station n to users within the base station's service area can be expressed as:

[0092]

[0093] Where P n W b N represents the transmission power and channel bandwidth of base station n, respectively. b Refers to the power spectral density of noise in a terrestrial network.

[0094] Note that the update scenario under consideration involves multiple time slots, h n This will change over time. Therefore, to ensure that file packets broadcast by edge caching devices via wireless channels in a space-ground integrated caching network can be successfully transmitted to users within a limited time, the file transmission success rate must be higher than a threshold δ, i.e., Pr(t) s2u ≤t0)≥δ, where t s2u Let ti represent the time required for the base station's caching device to deliver all cached encoded packets of a certain file, and t0 represent the maximum time allowed for a single file delivery. Therefore, according to equation (2), base station n represents the time required to successfully transmit all cached encoded packets of file i. The minimum power required is:

[0095]

[0096] in express inverse function, This represents the amount of data in the encoded packet of file i cached by base station n.

[0097] Satellite-User Link: Satellite communication channels are affected by a combination of factors, including free-space loss, antenna gain, shadowing fading, and many others. Energy loss and shadowing fading are the most significant aspects. Specifically, energy loss can be modeled using the Welb model; therefore, the energy loss in the satellite-user link can be modeled as follows:

[0098]

[0099] Among them G t G r These represent the antenna gains for the satellite and the user, respectively; λ refers to the electromagnetic wave wavelength; H... l f refers to the altitude of a satellite above the ground. rain For the parameters of the Welb distribution, the shadowed Rayleigh dispersion fading model, which considers both line-of-sight (LOS) shadowing and multipath shadowing, is frequently used in wireless channel environments with various frequency bandwidths. Therefore, |h s | 2 The cumulative distribution function can be expressed as:

[0100]

[0101] Where |h s | 2 Channel gain representing shadow fading. Where 2b represents the average power of the dispersion component, Ω represents the average power of the LOS loss component, and c represents the Nakagami fading coefficient.

[0102] Considering that user equipment in the ISTN can achieve direct communication with satellites, the total gain g of the satellite-user link is... s The combined effects of energy loss and shadow dispersion should be taken into account:

[0103] g s =G s |h s | 2 (6)

[0104] Therefore, similar to a base station-user link, the satellite buffers all encoded packets related to file i for successful transmission. The minimum power required is:

[0105]

[0106] in express inverse function, W represents the data size of the encoded packet in satellite cache file i. s N represents the satellite's transmission power and channel bandwidth, respectively. s This represents the power spectral density of noise in satellite communications.

[0107] Gateway-to-User Link: For the gateway-to-user link, to simplify the problem, only its downlink channel overhead is considered, and it is assumed that there is no packet loss during uplink forwarding. Therefore, similar to the satellite-to-user link, successful transmission... The minimum power required is:

[0108]

[0109] Where ε represents the processing and forwarding latency in the gateway-satellite shared link. This represents the amount of data that file i needs to forward in the gateway-user link, i.e.

[0110] Establish optimization problem

[0111] Based on the coordinated delivery of the three links, all user requests in the network can be guaranteed to be satisfied. Therefore, the amount of data transmitted in the network is equivalent to the total amount of user request data in the network, that is:

[0112]

[0113] in This represents the expected number of user requests during the k-th file request phase in both satellite period T and base station period t.

[0114] The system's energy consumption focuses only on the power consumption of downlink file transfers; therefore, the total system energy consumption can be expressed as the sum of the power costs of the three links, i.e.:

[0115]

[0116] in and These represent the power overhead derived during satellite period T and base station period t, respectively. P i sat and

[0117] This leads to the energy efficiency optimization problem in the space-ground integrated scenario:

[0118]

[0119] Where T is the index of the satellite update cycle, and let the satellite update cycle T contain W base station update cycles t, that is... Equation (12) represents the cache space limit of the base station, Equation (13) represents the cache space limit of the satellite, and Equation (14) represents that the maximum cached data per file in any cache device is sufficient to recover the data of the source file, so as to reduce the invalid cache in the cache and further improve the utilization rate of the cache.

[0120] Optimize problem decomposition:

[0121] Considering that in ISTN, due to the different update cycles of base stations and satellites, the energy efficiency optimization problem can be regarded as a long-short-term update problem. Therefore, problem P(1) can be decomposed into two sub-problems according to the time scale:

[0122]

[0123] st(12),(13),(14)(16) and

[0124]

[0125] st(12),(13),(14)(18)

[0126] To address this update problem, an iterative algorithm inspired by the coordinate descent (BCD) method was designed, which continuously follows these steps:

[0127] (1) Fixed Decision-making strategy, solve Make maximum.

[0128] (2) Fixed Decision-making strategy, solve Make maximum.

[0129] By continuously executing the above steps, the algorithm will gradually converge, yielding the optimal solution that maximizes energy efficiency.

[0130] DDPG Algorithm Design:

[0131] 1) Design of DDPG algorithm for terrestrial network (DDPG-SBS):

[0132] State: Consists of the current caching decision content and the predicted value of the user request information, that is:

[0133]

[0134] in Let be the predicted probability of file request for file i in cell n during satellite period T and base station period t.

[0135] Action: The action is determined by the proportion of cached data for each file stored in each SBS. Combining, that is:

[0136]

[0137] in

[0138] Rewards: Rewards are represented by the system's short-term energy efficiency, i.e.:

[0139]

[0140] 2) Design of DDPG algorithm for satellite networks (DDPG-SAT):

[0141] state:

[0142]

[0143] in Refers to state-based As input, the base station cache update decision for satellite update time slots is obtained using the DDPG-SBS algorithm. in

[0144] Action: An action is represented by the proportion of cached data for each file stored in the satellite, i.e.:

[0145]

[0146] Rewards: Rewards are represented by the system's long-term energy efficiency, i.e.:

[0147]

[0148] Behavioral cloning design:

[0149] Behavioral cloning technology was chosen for pre-training the policy network of the DDPG algorithm. By imitating expert policies, DDPG can quickly acquire initial decision-making capabilities, providing a higher starting point for subsequent training. To apply behavioral cloning technology to the DDPG algorithm training process, the input and output of the expert policy need to be rewritten in the form of state actions, allowing... This represents the caching decision returned by the expert policy, where the input states during the pre-training of the DDPG-SBS and DDPG-SAT algorithms are:

[0150]

[0151] The DDPG-SBS algorithm, DDPG-SAT algorithm, and satellite-ground cooperative cache update algorithm, which combine pre-training techniques, are shown in Tables 1, 2, and 3, where E sbs and E sat This represents the maximum number of training sessions.

[0152] Table 1 DDPG-SBS-BC Algorithm

[0153]

[0154]

[0155] Table 2 DDPG-SAT-BC Algorithm

[0156]

[0157] Table 3 DDPG-SBS-SAT-BC Algorithm

[0158]

[0159] Please see Figure 3 , Figure 3 This diagram illustrates a structural block diagram of a satellite-ground integrated cache update and cooperative transmission device according to an embodiment of the present invention. The device 400 includes:

[0160] The network system construction module 410 is used to construct a space-ground integrated caching network system, establish three cooperative transmission links based on the space-ground integrated caching network system: a base station-user link, a satellite-base station link, and a gateway-satellite shared link, and establish a channel model for these three cooperative transmission links based on a probability density function.

[0161] The objective optimization problem construction module 420 is used for the channel model, that is, based on the data transmission volume and power consumption of the three links, to establish an optimization problem P1 with the cache content of the satellite and base station layers as optimization variables and the goal of maximizing the long-term system energy efficiency.

[0162] The target optimization problem solving module 430 is used to solve the optimization problem P1 based on the deep deterministic policy gradient algorithm. Behavioral cloning technology is used to improve the training performance of the deep deterministic policy gradient algorithm. During the solution process, based on the asynchronous update mechanism of satellites and base stations, the optimization problem P1 is decomposed into a base station cache content update sub-problem P1.1 and a satellite cache content update sub-problem P1.2. Sub-problem P1.1 uses the deep deterministic policy gradient algorithm to obtain the base station cache decision policy under the current satellite cache content distribution, based on the predicted user request information within the base station update cycle. Sub-problem P1.2 uses the deep deterministic policy gradient algorithm to obtain the base station cache decision policy based on the current base station cache policy and the predicted user request information within the update cycle.

[0163] The algorithm yields the satellite caching decision strategy;

[0164] The caching strategy acquisition module 440 is used to converge the deep deterministic strategy gradient algorithm for the base station cache content update subproblem P1.1 and the satellite cache content update subproblem P1.2 through iterative training, and finally obtain the optimal satellite-ground integrated collaborative caching strategy.

[0165] In addition to the above module, the device 400 may also include other components; however, since these components are not relevant to the embodiments of this disclosure, their illustrations and descriptions are omitted here.

[0166] Other specific working processes of the space-ground integrated cache update and cooperative transmission device 400 are described in the above-described embodiment of the space-ground integrated cache update and cooperative transmission method, and will not be repeated here.

[0167] This invention also provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the space-ground integrated cache update and cooperative transmission method as described in any embodiment of this invention.

[0168] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed, implements the space-ground integrated cache update and cooperative transmission method as described in any embodiment of this invention.

[0169] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0170] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0173] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] To verify the effectiveness of the method of the present invention, simulation experiments were conducted:

[0175] The scenario parameters are as follows: Unless otherwise specified, consider a multi-base station distribution scenario with 6 base stations, where the file library size is 20, the SBS cache size is set to 10% of the total file library data, and the satellite cache size is set to 20%.

[0176] Base station layer DDPG algorithm parameters: A five-layer fully connected network is used to build the actor network, including one input layer, three hidden layers, and one output layer, with each layer having a dimension of 800. The critic network also uses a five-layer fully connected network, but to improve convergence speed, the dimensions from the input layer to the output layer are 800, 400, 200, 100, and 1 respectively. The learning rate for the actor network is maintained at 0.00001, and the learning rate for the critic network is maintained at 0.001. Cache pool R sbs We chose a capacity of 10,000, a batch size of 32, a soft update parameter ρ of 0.005, a discount factor γ of 0.99, and a pre-training session size of 500.

[0177] Satellite DDPG algorithm parameters: Similarly, the same parameters as the base station layer DDPG are used to build the satellite actor network and critic network, and the learning rate and other parameters are also kept consistent. Other simulation parameters are shown in Table 4.

[0178] Table 4 Simulation Parameter Table

[0179]

[0180] Simulation comparison and analysis:

[0181] Uniform Caching (UC): The caching device evenly distributes the amount of encoded data for each file to fill its cache space.

[0182] Most Popular File Placement (MP): The cache device is based on predicted values. Each file is stored in a relative proportion.

[0183] DDPG: The DDPG-SBS-BC and DDPG-SAT-BC algorithms, which do not include the behavior cloning pre-training step, are used as baseline algorithms and named DDPG-SBS and DDPG-SAT, respectively.

[0184] Trust Region Constrained Optimization Algorithm (TR): This algorithm is considered the most ideal algorithm to determine the upper bound of the proposed algorithm's performance.

[0185] Algorithm combination: It is described in the format of "algorithm-base station layer / satellite-pre-training technology". For example, DDPG-SBS-BC+UC-SAT means that the DDPG-SBS-BC algorithm, which has been pre-trained by behavioral cloning, is applied at the base station layer, while the evenly distributed placement algorithm is adopted at the satellite layer.

[0186] Figure 4 This paper demonstrates the optimized performance of the DDPG-SBS and DDPG-SAT algorithms, with two base stations and a file library containing five files. The cache capacity of the base stations and satellites represents 25% and 50% of the total file library, respectively. This experiment does not consider the variability of environmental parameters and real-time request information, and can be considered a pure cache allocation problem. Unlike the optimization scenario of the traditional DDPG algorithm, ISTN involves joint decision-making between two layers of cache devices. Therefore, when updating the allocation of one layer of cache devices, the influence of the caching strategy of the other layer must be considered. The results show that regardless of whether the other layer adopts the most popular file placement strategy or the evenly distributed placement strategy, the proposed DDPG-SBS and DDPG-SAT algorithms can converge within 800 episodes. Compared with the MP-SBS+UC-SAT algorithm, the two algorithms achieve performance improvements of 9% and 4%, respectively. Furthermore, compared with the TR algorithm, the performance of the DDPG-SBS algorithm is very close to the TR level, while the DDPG-SAT algorithm achieves 70% of the TR algorithm's gain.

[0187] Figure 5This paper presents a performance comparison of algorithms such as DDPG-SBS-SAT-BC under different file library sizes. The most popular file placement strategy is used as the expert strategy imitated by the base station DDPG algorithm, while the even distribution strategy is used as the expert strategy imitated by the satellite. The results show that as the total number of files in the file library increases, the energy efficiency optimization level of each decision algorithm generally decreases. This is because the storage space of the edge caching device is limited, making it difficult to meet the caching needs of more files. Notably, when the total number of files in the file library is low (i.e., the number of files is less than or equal to 10), the most popular file placement strategy at the satellite layer achieves higher energy efficiency optimization compared to the even distribution scheme. However, as the total number of files in the file library increases, the energy efficiency performance of the even distribution strategy gradually surpasses that of the most popular file placement strategy. This phenomenon is consistent with the analysis of base station cache space. When the number of files is large, the even distribution strategy can allocate cache resources more evenly, thereby improving overall energy efficiency.

[0188] from Figure 5 As can be seen, compared to the UC-SBS+UC-SAT algorithm combination, the DDPG-SBS-SAT-BC, DDPG-SBS-BC+UC-SAT, and MP-SBS+UC-SAT algorithm combinations achieved performance improvements of 49%–59%, 31%–48%, and 26%–30%, respectively, demonstrating the significant advantages of these optimization algorithms under different conditions. Overall, as the file library size increases, the DDPG-based algorithms exhibit strong adaptability and are better able to handle complex caching optimization problems.

[0189] Figure 6 The performance of algorithms such as DDPG-SBS-SAT-BC is demonstrated under different numbers of base stations. It can be seen that, compared to the UC-SBS+UC-SAT algorithm combination, the algorithm combinations DDPG-SBS-SAT-BC, DDPG-SBS-BC+UC-SAT, and MP-SBS+UC-SAT maintain stable optimization capabilities under different numbers of base stations, achieving performance improvements of approximately 53%, 46%, and 28%, respectively.

[0190] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A satellite-ground integrated cache update and cooperative transmission method, characterized in that, The method includes the following steps: Construct a space-ground integrated caching network system, and establish three cooperative transmission links based on the space-ground integrated caching network system: base station-user link, satellite-base station link, and gateway-satellite shared link. Establish a channel model for these three cooperative transmission links based on the probability density function. Based on the channel model, an optimization problem P1 is established with the cache contents of satellite and base station layers as optimization variables and the goal of maximizing long-term system energy efficiency. The optimization problem P1 is solved using the Deep Deterministic Policy Gradient Algorithm (DPRQA), and behavioral cloning techniques are employed to improve the training performance of DPRQA. During the solution process, based on the asynchronous update mechanism of satellites and base stations, optimization problem P1 is decomposed into two sub-problems: base station cache content update P1.1 and satellite cache content update P1.

2. Sub-problem P1.1 uses the DPRQA to obtain the cache decision strategy under the current base station cache content distribution, based on the predicted user request information within the base station update cycle. Sub-problem P1.2 uses the DPRQA to obtain the satellite cache decision strategy based on the current base station cache strategy and the predicted user request information within the update cycle. Through iterative training, the deep deterministic policy gradient algorithm for updating base station cache content subproblem P1.1 and satellite cache content subproblem P1.2 converges, ultimately obtaining the optimal satellite-ground integrated cooperative caching strategy.

2. The method according to claim 1, characterized in that, The specific implementation steps of the method include: Step 1: Based on the constructed satellite-ground integrated caching network system, calculate the system energy efficiency expression using channel parameters; Step 2: Initialize the network parameters for the deep deterministic policy gradient algorithm at the satellite and base station layers; Step 3: Collect a segment of historical user request information, select expert strategies to generate corresponding base station layer expert caching schemes and satellite expert caching schemes, assemble them into {user request information, expert caching scheme}, and store them in the base station layer expert experience cache pool and the satellite expert experience cache pool respectively. Step 4: Randomly select a set of samples from the base station layer expert experience cache pool and the satellite expert experience cache pool respectively, and pre-train the Actor network in the deep deterministic policy gradient algorithm using the backpropagation mechanism in supervised learning; Step 5: Determine if the number of training rounds exceeds the set number of expert training rounds. If not, repeat step 3. If the set number of expert training rounds is reached, the Actor network is considered to be pre-trained and the parameters of the Actor network are copied to the target Actor network. Step 6: Determine whether the current stage is satellite update or base station update. If it is satellite update, proceed to step 7; if it is base station update, proceed to step 11. Step 7: Collect the predicted values ​​of user request information in the current satellite phase, as well as the network parameters of the current satellite depth deterministic strategy gradient algorithm, satellite experience cache pool, and base station cache decision strategy. Step 8: Generate a base station caching scheme based on the current base station caching decision strategy and the predicted value of user request information, and combine {current satellite caching scheme, base station caching scheme, user request information} into the input state of the satellite depth deterministic policy gradient algorithm, obtain the output action, and observe the long-term network system energy efficiency as a reward after taking the action; Step 9: Put the experience samples {state, action, reward, next state} into the satellite experience cache pool, and randomly select a set of samples from it to train the deep deterministic policy gradient algorithm, and update the network parameters of the deep deterministic policy gradient algorithm. Step 10: Determine if the training count has reached the set maximum number of times. If not, repeat step 8. If the set maximum number of times has been reached, the current satellite depth deterministic policy gradient algorithm is considered to have been trained, the current satellite cache decision content is output, and the process jumps to step 15. Step 11: Collect the predicted values ​​of user request information in the current base station phase, as well as the network parameters of the current base station deep deterministic strategy gradient algorithm, the base station experience cache pool, and the satellite caching scheme; Step 12: Combine {current satellite caching scheme, base station caching scheme, user request information} into the input state of the base station deep deterministic policy gradient algorithm, obtain the output action, and observe the short-term network system energy efficiency as a reward after taking the action. Step 13: Put the experience samples {state, action, reward, next state} into the base station experience cache pool, and randomly select a set of samples from it to train the deep deterministic policy gradient algorithm, and update the network parameters of the deep deterministic policy gradient algorithm. Step 14: Determine whether the training count has reached the set maximum number of times. If not, repeat step 11. If the set maximum number of times has been reached, it is determined that the training of the current base station deep deterministic policy gradient algorithm is complete. Output the current base station cache decision content and jump to step 15. Step 15: Determine if there is no new update phase. If there is a new update phase, repeat step 6; otherwise, end.

3. The method according to claim 1, characterized in that, The optimization problem P1, which aims to maximize the long-term system energy efficiency, is specifically expressed as: P(1): s.t.C1: C2: C3: Where T is the index of the satellite update cycle, and let the satellite update cycle T contain W base station update cycles t, that is... Refers to the cache matrix of base station n within the satellite update cycle T and the base station update cycle t. This represents the cache vector of satellites within the base station update period t within the satellite update period T. This represents the total amount of user request data in the network. This represents the total energy consumption of the system. k re To the extent that the amount of data in the source file is sufficient to recover, C1 represents the cache space for file i, and C1 represents the cache space limit of the base station. C1 represents the maximum buffer space of the base station, and C2 represents the buffer space limit of the satellite. C3 represents the maximum cache space of the satellite, and C3 represents the maximum amount of cached data for a file in any cache device that is sufficient to recover the original file. N is the number of base stations. All file requests within the defined scenario belong to a file library containing F files.

4. The method according to claim 3, characterized in that, The expressions for the base station cache content update subproblem P1.1 and the satellite cache content update subproblem P1.2 are as follows: P1.1: stC1,C2,C3 P1.2: stC1,C2,C3.

5. The method according to claim 4, characterized in that, The solution to the base station cache content update subproblem P1.1 and the satellite cache content update subproblem P1.2 involves repeatedly performing the following steps until the algorithm gradually converges: fixed Decision-making strategies, solving Make maximum; fixed Decision-making strategies, solving Make maximum.

6. The method according to claim 1, characterized in that, The base station caching decision strategy under the current satellite cache content distribution is obtained using the deep deterministic policy gradient algorithm, including: Base station status: Consists of the current satellite cache decision content and the predicted value of user request information, i.e.: in Let be the predicted probability of file request for file i in cell n during satellite period T and base station period t. The cache vector of satellites within the base station update period t, which is within the satellite update period T. The cache matrix of the base station within the previous update cycle t-1 within the satellite update cycle T; Base station actions: Actions are determined by the proportion of cached data for each file stored in each base station. Combining, that is: in Base station incentive: The incentive is represented by the system's short-term energy efficiency, i.e.: This represents the total amount of user request data in the network. This indicates the total energy consumption of the system.

7. The method according to claim 1, characterized in that, The satellite caching decision strategy is obtained using the deep deterministic policy gradient algorithm, including: Satellite status: in Represents state-based As input, the base station cache update decision for satellite update time slots is obtained using the output of the deep deterministic policy gradient algorithm. This represents the cache vector of the base station in the last base station update cycle within satellite cycle T-1. This represents the cache vector of satellites within the base station update period t within the satellite update period T-1. in The mean of the predicted file request probability values; Satellite actions: Actions are determined by the proportion of cached data for each file stored in the satellite. Combining, that is: Satellite rewards: Rewards are represented by the system's long-term energy efficiency, i.e.: This represents the total amount of user request data in the network. This indicates the total energy consumption of the system.

8. A satellite-ground integrated cache update and collaborative transmission device, characterized in that, The device includes: The network system construction module is used to construct a space-ground integrated caching network system. Based on the space-ground integrated caching network system, three cooperative transmission links are established: a base station-user link, a satellite-base station link, and a gateway-satellite shared link. A channel model is established for these three cooperative transmission links based on a probability density function. The objective optimization problem construction module is used to establish an optimization problem P1 based on the channel model, with the cache contents of the satellite and base station layers as optimization variables and the objective of maximizing the long-term system energy efficiency. The target optimization problem solving module is used to solve the optimization problem P1 based on the deep deterministic policy gradient algorithm. Behavioral cloning technology is employed to improve the training performance of the deep deterministic policy gradient algorithm. During the solution process, based on the asynchronous update mechanism of satellites and base stations, the optimization problem P1 is decomposed into two sub-problems: base station cache content update P1.1 and satellite cache content update P1.

2. Sub-problem P1.1 uses the deep deterministic policy gradient algorithm to obtain the cache decision strategy under the current base station cache content distribution, based on the predicted user request information within the base station update cycle. Sub-problem P1.2 uses the deep deterministic policy gradient algorithm to obtain the satellite cache decision strategy based on the current base station cache strategy and the predicted user request information within the update cycle. The caching strategy acquisition module is used to converge the deep deterministic strategy gradient algorithm for the base station cache content update subproblem P1.1 and the satellite cache content update subproblem P1.2 through iterative training, and finally obtain the optimal satellite-ground integrated collaborative caching strategy.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the space-ground integrated cache update and cooperative transmission method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the space-ground integrated cache update and cooperative transmission method as described in any one of claims 1-7.

Images