Dynamic cache updating method for satellite-ground convergence network

Abstract

The invention provides a dynamic cache updating method for a satellite-ground fusion network, and the method comprises the steps: 1, building an LEO satellite-base station time-varying incidence matrix according to the position information of an LEO satellite, and building a ground base station-LEO satellite-core network gateway three-layer cache architecture network model based on the time-varying incidence matrix; 2, constructing a multi-content sub-library model to represent content popularity under different satellite coverage areas; 3, constructing a communication link model of a satellite and a base station in the network, and establishing a performance index for measuring the quality of a satellite-ground caching strategy; and step 4, proposing a satellite-ground and inter-satellite adaptive threshold cooperative caching strategy based on meta reinforcement learning, learning change characteristics of content popularity among different coverage areas, and realizing caching updating during area switching based on content characteristic similarity among satellite coverage areas and an alternative content index set at a satellite node. The method has the beneficial effects that the generalization ability of deep reinforcement learning is improved, and the network energy efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of edge cache technology, and in particular to a dynamic cache update method for a satellite-ground fusion network. Background Art

[0002] With the development of communication technology, wireless communication has brought users more efficient and convenient network services, accompanied by an explosive growth of data content in the network. It is predicted that mobile data traffic is expected to grow at a compound annual growth rate of 23% between 2023 and 2030. By the end of 2030, the penetration rate of mobile users is expected to reach 74% of the global population. By then, the number of global mobile users is expected to increase to 6.3 billion, and data traffic will reach more than 465EB per month. The unprecedented growth of mobile data traffic has brought great challenges to traditional cellular networks in providing high throughput and multi-access services under limited backhaul link capacity. Due to the limitations of ground network spectrum resources and energy efficiency ceiling, traditional ground networks with cache cannot cope with the expected sharp growth of data. Satellite networks have the advantages of broadcast transmission and huge coverage. Therefore, the concept of STIN, which combines ground networks and satellite networks, has been proposed, and satellite networks are regarded as a supplement to ground networks to fill the network's shortcomings in cache content transmission. However, the resources of satellite networks are very limited, and the onboard payloads are also constrained by hardware conditions, such as limited spectrum and limited communication energy, which makes it difficult to increase the transmission capacity of the network. At the same time, STIN faces many difficulties in processing a large number of content requests from users and is under great pressure to ensure service quality. Therefore, how to provide high-quality content delivery services has become a key issue that has attracted widespread attention.

[0003] Most existing studies consider the pre-known content popularity to characterize the request preferences of ground users. Since user request preferences vary in the spatial dimension due to the dynamic coverage of satellites over geographical areas, and also show dynamic changes with different time intervals, that is, there are differences in the time dimension, the constant content popularity model is likely to cause a mismatch between theoretical and practical results. Therefore, traditional caching strategies are not enough to provide good edge caching performance. On the other hand, most existing studies use a centralized approach to update caching decisions, such as using medium earth orbit (MEO), geostationary earth orbit (GEO) satellites or ground stations as centralized controllers to make caching decisions for all nodes in the network. The centralized approach uses a single controller, which may cause unreliable caching decision update processes and generate additional communication overheads, as well as large computational overheads, given the highly dynamic connection characteristics and limited coverage of STIN. However, considering the heterogeneity of network nodes, each node only observes the local environment. In this case, a fully distributed structure is difficult to fully demonstrate the advantages of collaboration between nodes. In view of this, when updating the node cache status in STIN, the information gap can be eliminated by synchronizing the information between nodes, and then the cache status update work can be carried out in a distributed manner, so as to take into account the local characteristics of each node and the overall collaboration needs. Summary of the invention

[0004] In order to solve the problems in the prior art, the present invention provides a dynamic cache update method for a satellite-ground fusion network, comprising the following steps: Step 1: establishing a time-varying association matrix of a LEO satellite-base station according to the position information of a LEO satellite, and constructing a three-layer cache architecture network model of a ground base station-LEO satellite-core network gateway based on the time-varying association matrix of a LEO satellite-base station;

[0005] Step 2: Construct a multi-content sub-library model to characterize the content popularity in different satellite coverage areas; each content sub-library contains multiple contents, representing a certain type of content. Users in different regions will prefer content in different sub-libraries. The content popularity distribution in each region follows the Zipf distribution. There are differences in preferences between different regions, and the content of some sub-libraries is only popular in some regions.

[0006] Step 3: Build a communication link model between satellites and base stations in the network and establish performance indicators to measure the quality of satellite-to-ground caching strategies;

[0007] Step 4: First, a satellite-ground and inter-satellite adaptive threshold collaborative caching strategy based on meta-reinforcement learning is proposed to quickly learn the changing characteristics of content popularity between different coverage areas, and then based on the similarity of content features between satellite coverage areas and the set of alternative content indexes at satellite nodes, fast cache updates are achieved during area switching.

[0008] As a further improvement of the present invention, in the step 1, it also includes:

[0009] Step 1: Use a size S J ×B I The matrix G t To describe the association state between the satellite and the base station in each time slot, it is expressed as:

[0010]

[0011] in Indicates that the satellite S in time slot t j With base station B i The connection relationship of satellite S j With base station B i In the associated state, 0 is the opposite;

[0012] Step 2: Deploy caches on the base station side and the satellite side. The cache capacity of each base station is The cache capacity of each satellite is The cache status of the base station is represented as The cache status of the satellite is represented by and Respectively represent base station B i and LEO satellite S j The cth content stored in the cache.

[0013] As a further improvement of the present invention, the step 2 is specifically: dividing the content set into M content sub-libraries to represent different types of content, and the set is represented as:

[0014]

[0015] Users in each region may be interested in the contents in one or more content sub-libraries. There are multiple files in the database, and the popularity ranking of each content will change over time, but the probability of the content in each sub-library being requested by the user always follows the probability P m , expressed as:

[0016]

[0017] Content popularity It follows the Zipf distribution, where i is the popularity ranking of the content;

[0018] In step 2, time is divided into multiple discrete time slots, represented as t=1, 2, ..., T. In each time slot t, users in the area will generate content requests based on content preferences. j The upper limit of the number of user requests received is the sum of the number of users in each coverage area;

[0019] In step 2, the method of transmitting content to users uses a layered service architecture. First, the local base station responds. If there is no corresponding content, the request is forwarded to the associated satellite. Finally, the remote gateway provides the corresponding content. The base station and the LEO satellite update the cached content through the gateway connected to the remote core network.

[0020] As a further improvement of the present invention, the step three comprises:

[0021] Step 1: Model the downlink feeder link, uplink feeder link and ground wireless transmission link, and obtain the unit content transmission delay τ of the base station respectively b , the unit content transmission delay τ of LEO satellite s , the unit content transmission delay τ of the gateway g , and the uplink feeder link power loss P UL and ground wireless propagation link power loss P BS ;

[0022] Step 2: Based on the cache architecture network model of step 1 and the multi-content sub-library model of step 2, the delay reduction gain of the ground base station and the LEO satellite is calculated;

[0023] Step 3: Calculate the symmetric difference of the cache states of the two time slots before and after to obtain the number of updated contents. The expression is:

[0024] D t =|x t Δ x t-1 | (22)

[0025] Then, we get the LEO satellite S j and ground base station collection The cache cost calculation formula is:

[0026]

[0027] Then use P local,t and E local,t Represents the cache cost and utility of the local network to which the current agent belongs:

[0028]

[0029] Among them, ξ represents the proportionality coefficient, Indicates the cache status of the base station. Indicates the cache status of the satellite.

[0030] As a further improvement of the present invention, step 2 specifically includes: firstly defining a function I(x,y) for determining whether the element x is in the vector y, the expression is:

[0031]

[0032] Base Station B i User requests received at time slot t The cache status is This gives the base station B i The delay reduction gain is:

[0033]

[0034] according to Derive with satellite S j Collaborative base station collection After the user request is responded to by the service of the base station layer, the unanswered content request is forwarded to the satellite S j ,Right now From formula 11, we can get:

[0035]

[0036] LEO satellite cache status is Thus, we get the satellite S j The delay reduction gain is:

[0037]

[0038] Then, we get the satellite S in time slot t j Integration with base station The delay reduction gain is:

[0039]

[0040] As a further improvement of the present invention, in step 4, in the meta-training of meta-reinforcement learning, the outer loop is from the task set Sample a batch of tasks Tasks are defined as user requests generated based on the popularity changes of different types of content, simulating the popularity differences between regions that the satellite will face. The inner loop uses each task obtained by sampling. To train the model, we obtain K experience trajectories for each task through the inner loop, denoted as τ j ={(s j,0,a j,0 ,r j,0 ,s j,1 ),(s j,1 ,a j,1 ,r j,1 ,s j,2 ),...,(s j,K-1 ,a j,K-1 ,r j,K-1 ,s j,K )}, and finally the outer loop performs cross-task learning based on the experience trajectory collected by the inner loop to update the meta-parameters of the agent and

[0041] As a further improvement of the present invention, in step 4, when the satellite needs to switch regions, the satellite in the next region will accumulate the request status. Sent to satellite S j , the current satellite will calculate the Manhattan distance between the cumulative request state vectors of the two regions as a measure of the difference in content characteristics between the regions. The expression is:

[0042]

[0043] It reflects the difference in content characteristics between the two regions. The biggest difference is that the user request states of the two regions are completely different. The difference value at this time is recorded as W max ,when When υ∈[0,1], υ is a proportional parameter, the satellite's action tends to explore, and as the reward value increases, exploration is reduced. f represents the user's request, f represents the content, The meaning is that the satellite S j The associated base station is at t 0 The number of requests for different contents received from users during a period from t to t is represented by a vector.

[0044] As a further improvement of the present invention, in step 4, using the set Indicates a set of candidate content. The corresponding index is stored on the satellite side. The upper limit of the size is defined as F alt , the LEO satellite stores the corresponding content indexes of the user requests received in the past in the collection Then the first-in-first-out algorithm is used as the collection The update algorithm of When the size reaches the upper limit, the content index that entered the collection earliest is eliminated first.

[0045] As a further improvement of the present invention, the MATD3 algorithm is used in the inner loop of meta-training to perform small sample learning on different distributions. The specific steps are as follows:

[0046] Step S1: Initialize meta-strategy parameters and critic network parameters θ π ,θ 1 and θ 2 , discount factor γ, outer learning rate η, inner learning rate ε, task distribution

[0047] Step S2: Each task sampled using the outer loop Train the model;

[0048] Step S3: Using strategy parameters Collecting experience trajectories τ j ;

[0049] Step S4: Calculate the reward R on the experience trajectory using formula (30), and calculate the critic network loss function according to formula (32);

[0050] Step S5: Through stochastic gradient descent Update the parameter θ 1 and θ 2 , ε represents the inner learning rate, represents the loss function, Represents the network parameters in the inner loop;

[0051] Step S6: According to formula (33) and Calculate the gradient to update the parameter θ π , represents the policy parameters, represents the gradient, ε represents the inner learning rate;

[0052] Step S7: Soft update the target network parameters, where k = 1, 2, is the meta-parameter of the agent, is the parameter of the target network, because there are two Critic networks, divided into Critic 1 network and Critic 2 network, they have their own network parameters, corresponding to θ 1 and θ 2 , the k here represents 1 and 2.

[0053] As a further improvement of the present invention, the Reptile method based on meta-learning is used to train the network model, and the expression is:

[0054]

[0055] where m is is the number of tasks in the network, η is the outer learning rate, a is the actor network, and c is the critic network.

[0056] The beneficial effects of the present invention are as follows: the dynamic cache update method of the present invention fully considers the inter-satellite and satellite-ground collaboration in scenarios with highly heterogeneous content popularity, uses meta-learning to improve the generalization ability of deep reinforcement learning, uses multi-agent reinforcement learning to improve the collaboration of distributed systems, and makes full use of the local environmental information observed by the agent to construct an adaptive threshold, effectively improving network energy efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of the dynamic cache update method of the present invention;

[0058] Figure 2 is the STIN network model of the present invention;

[0059] Figure 3 It is a schematic diagram of user content preference area heterogeneity according to the present invention. DETAILED DESCRIPTION

[0060] In order to improve the service quality of the network under limited cache resources, the present invention carries out research on multi-node collaborative caching technology for Satellite-Terrestrial Integrated Network (STIN) in the scenario of spatiotemporal heterogeneity of content popularity. First, considering the spatiotemporal heterogeneity of content popularity, the cache optimization strategy of a sub-network consisting of a satellite and multiple base stations over a period of time is mainly considered. Then, considering the dynamic nature of the connection between satellites and base stations, as well as the richness of data content and the regional heterogeneity of popularity, the satellite-to-ground and inter-satellite collaborative caching strategies in multi-satellite scenarios are studied to alleviate the pressure on the return link, reduce service latency, and improve the overall network energy efficiency.

[0061] In the scenario of satellite-ground fusion network, the present invention aims at the cache strategy design problem of satellites and base stations and provides an algorithm for collaborative cache strategy design based on Meta-RL. The purpose is to design a distributed cache strategy for satellites and base stations when STIN provides content services to users under the condition of highly heterogeneous content popularity, thereby improving the generalization ability of the cache strategy and reducing the service delay and cache update cost of the network.

[0062] like Figure 1 As shown, the present invention discloses a dynamic cache update method for a satellite-ground fusion network, comprising the following steps:

[0063] Step 1: According to the position information of LEO (low earth orbit) satellite, a time-varying association matrix of LEO satellite and base station is established. Based on the time-varying association matrix of LEO satellite and base station, a three-layer cache architecture network model of ground base station, LEO satellite and core network gateway is constructed. Due to the limited cache capacity of satellites and base stations, a cache model of satellites and base stations is constructed, and the corresponding content of the cache is represented by maintaining a fixed-size content set.

[0064] Step 2: Construct a multi-content sub-library model to characterize the content popularity in different satellite coverage areas; Since the amount of data content in the real world is massive and scattered in different regions, each content sub-library contains multiple contents, representing a certain content. Users in different regions will prefer content in different sub-libraries, and the content popularity distribution in each region follows the Zipf distribution. There are large differences in preferences between different regions, and the content of some sub-libraries is only popular in some regions.

[0065] Step 3: Construct the communication link model between the satellite and the base station in the network, and then establish performance indicators to measure the quality of the satellite-ground cache strategy. The performance indicators mainly include content transmission delay, cache update cost and network energy efficiency. When the user sends a content request to the base station, the base station first queries whether the requested content exists in the local cache. When the content is cached locally, the base station can directly send the content to the user with a lower delay. At this time, the content transmission delay only includes the transmission delay from the base station to the user. At the same time, this situation is also called a cache hit. If the base station cannot find the corresponding content when querying the local cache, it queries the cache of the associated satellite. When the content requested by the user is cached in the associated satellite, the associated satellite sends the corresponding content to the user; if the base station cache and its associated satellite do not cache the content requested by the user, it is necessary to obtain the content from the core network server to serve the user, which will cause a large delay. For these three service user situations, the corresponding content transmission delay is calculated respectively, and the network service delay can be obtained. The satellite and base station update the cache content respectively relying on the uplink feeder link and the ground return link. The uplink feeder link mainly considers free space propagation loss and assumes that the base station is in a remote access network. Both the satellite and the base station will generate non-negligible energy loss when updating the cache, which is regarded as the cache update cost.

[0066] Step 4: First, a satellite-ground and inter-satellite adaptive threshold collaborative caching strategy based on meta-reinforcement learning (Meta-RL) is proposed to quickly learn the changing characteristics of content popularity between different coverage areas. Then, based on the similarity of content features between satellite coverage areas and the set of alternative content indexes at satellite nodes, fast cache updates are achieved during area switching.

[0067] Due to the dynamics of satellites and the high heterogeneity of content popularity between regions at a large time scale, the satellite's caching strategy will continue to face changing content popularity distributions. If the satellite retrains the neural network model before arriving at a new area, it will require a large number of training rounds. Therefore, Meta-RL is used to quickly learn the changing characteristics of content popularity between different coverage areas. The Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3) algorithm is used in the inner loop of meta-training to perform small sample learning on different distributions, and cross-task learning is achieved in the outer loop. Subsequently, the same-orbit satellite collaboration is used to calculate the difference in content popularity between the two regions before and after the LEO satellite switches regions, determine the similarity of content features between the two regions, and build a terminal-side alternative content index set for the LEO satellite, reduce the dimension of the action space, and achieve fast cache updates during region switching.

[0068] Specifically, the algorithm models the collaborative caching strategy optimization problem as a Markov decision process and constructs a reward function based on the objective function. The reward function is constructed based on the local information of each cache node and the global information of the network to minimize the service delay and cache update cost of users in the system and maximize the network energy efficiency, that is, the system reward. The algorithm first improves the generalization ability of the model based on model-agnostic meta-learning (MAML), so that the edge nodes can flexibly adapt to the changing state of the popularity of various contents, and completes the collaborative caching strategy optimization based on MATD3. By using information sharing between satellites in the same orbit, the difference in content popularity between the two regions before and after the LEO satellite switches regions is calculated to determine the similarity of the content features of the two regions. In order to reduce the spatial dimension of action selection and be more in line with the actual situation, a satellite-side content index library with less content than the information source and in line with the needs of the satellite service area is constructed, thereby achieving improved network energy efficiency and rapid convergence of the caching strategy.

[0069] The present invention considers the algorithm design of satellite-ground and inter-satellite collaborative caching strategy based on meta-reinforcement learning in the satellite-ground fusion network scenario:

[0070] 1. System Model

[0071] (1) Network Model

[0072] The research scenario of the present invention includes multiple areas, such as Figure 2 As shown in Figure 2, a network model is constructed based on this scenario. The base station set is represented as The LEO satellite set is represented as In order to maintain the orbital altitude, LEO satellites orbit the earth at a speed of about 7.8 km / s, and the time for one orbit is usually between 1.5 and 2 hours. During the movement, LEO satellites will only cooperate with ground base stations within their coverage area to provide services to users, that is, the current satellite S j The area covered by satellite S j+1 Therefore, the relationship between the LEO satellite and the ground base station is constantly changing. The present invention uses a size S J ×B I The matrix G t To describe the association state between the satellite and the base station in each time slot, it is expressed as:

[0073]

[0074] in Indicates that the satellite S in time slot t j With base station B i The connection relationship of satellite S j With base station B i In order to compare the similarity of user preferences between different regions, LEO satellites can communicate with associated ground base stations as well as satellites one hop away from the same orbit.

[0075] Caches are deployed on the base station side and the satellite side. The cache capacity of each base station is The cache capacity of each satellite is Since the network service scope of the present invention is expanded from local areas to the orbital coverage area of ​​satellites, the coverage of each satellite does not overlap with that of other satellites on the same periodic orbit, which means that each user can only be served by one satellite at any time. Since user preferences in different regions are different, the data content is complex and the amount is huge, when a satellite enters an area where user preferences are greatly different from before, there will be a "blind spot" for the cache strategy, which cannot be predicted in advance. Therefore, the cache status of the base station and the satellite is expressed as and and Respectively represent base station B i and LEO satellite S j The cth content stored in the cache.

[0076] There is no need to consider the connection relationship between the satellite and the ground gateway separately, because the time-varying association matrix between the satellite and the base station includes the association relationship with the ground gateway.

[0077] (2) Content Request Model

[0078] Since LEO satellites need to orbit the earth at a high speed of about 7.8 km / s, they will periodically provide content services to different target areas along their orbits. As the satellites orbit, they will periodically provide content services to different target areas along their orbits. The popularity of content is often strongly correlated with the regional location. Figure 3 As shown in the figure, for the convenience of representation, regular hexagons are used to represent coverage cells, and a satellite coverage area consists of multiple cells. This spatial diversity is simulated by dividing the coverage area along the track into N areas with different content popularity, where different colors represent different user preferences for content. Note that a large satellite beam may cover multiple areas with different local content popularity distributions.

[0079] In the real world, network data content accumulates over time, but many of them become "cold content" over time, that is, content that is rarely or no longer requested by users in the recent period. Therefore, the present invention reasonably assumes that all data content requested by users at the moment is stored in the cloud and can be obtained through the core network. The content library is represented as a collection In order to conveniently represent content types and user preferences in different regions, the present invention divides the content set into M content sub-libraries to represent different types of content. The set can be expressed as:

[0080]

[0081] Users in each region may be interested in the content in one or more content sub-libraries. There are multiple files in the database, and the popularity ranking of each content will change over time, but the probability of the content in each sub-library being requested by the user always follows the probability P m , expressed as:

[0082]

[0083] The popularity of the content of this invention It follows the Zipf distribution, where i is the popularity ranking of the content.

[0084] Time is divided into multiple discrete time slots, denoted as t=1,2,…,T. In each time slot t, users in the area generate content requests based on their content preferences. Assume that the number of users in the coverage area of ​​each base station is U i , each user will initiate a request in a time slot, and base station B i The received user request is represented as in Indicates the content requested by user u, which may belong to the content library According to the satellite-base station time-varying correlation matrix G t , the cooperative relationship between LEO satellite and base station is expressed as Can be expressed with satellite S j The number of cooperating base stations, that is, the number of covered areas. Satellite S j The upper limit of the number of user requests received is the sum of the number of users in each coverage area, which can be expressed as Therefore, the user request received by the satellite is expressed as in Represents user u i The content of the request.

[0085] The way to transmit content to users uses a layered service architecture. First, the local base station responds. If there is no corresponding content, the request is forwarded to the associated satellite, and finally the remote gateway provides the corresponding content. The base station and LEO satellite can update the cached content through the gateway connected to the remote core network.

[0086] (3) Communication model

[0087] This communication model mainly studies the information transmission caused by the content transmitted between users, base stations and satellites, as well as the information transmission caused by the content transmitted between base stations, satellites and core networks and the update of cache status. Therefore, the links involved include the ground wireless transmission link between users and base stations, the feeder link between satellites and ground stations, and the backhaul link from subnetworks to core networks.

[0088] The loss of the downlink feeder link of the LEO satellite mainly considers the free space propagation loss and the atmospheric loss. The height of the LEO satellite is regarded as the distance between the satellite and the user. The specific expression is:

[0089]

[0090] Since free space propagation loss accounts for the main part of the downlink feeder link loss, and the atmospheric loss is mainly caused by the absorption and scattering of gas molecules in the earth's atmosphere, as well as meteorological factors such as rain attenuation and fog attenuation, the downlink link loss expression is:

[0091] L atm =A gas +A rain +A fog +... (41)

[0092] L DL =L FS +L atm (42)

[0093] The received signal-to-noise ratio of the downlink feeder link can be obtained as follows:

[0094] SNR DL (dB) = EIRP-L DL +G r -kT total -B s (43)

[0095] Where EIRP is the satellite’s equivalent isotropic radiated power, G r is the ground station receiving antenna gain, T total is the total noise temperature, B s is the satellite channel bandwidth. The Shannon formula can be used to obtain the maximum transmission rate of LEO satellite to transmit unit content to users, which is expressed as:

[0096] R DL =B s log(1+SNR DL ) (44)

[0097] In addition to free space propagation loss, ground wireless transmission links also need to consider large-scale fading and small-scale fading. Large-scale fading is usually described by the path loss exponential model, and the specific expression is:

[0098]

[0099] Where n is the path loss index, and d is the distance between the base station and the user. Common small-scale fading includes Rayleigh fading and Rice fading. Here, Rayleigh fading is considered, and the amplitude of the received signal obeys the Rayleigh distribution. The specific expression is:

[0100] P r,R =|h 2 P r,LS (46)

[0101] Where h is the fading factor. In this way, the received power after fading can be obtained, and the maximum transmission rate from the base station to the user can be calculated by Shannon's formula:

[0102]

[0103] Among them B b is the channel bandwidth of the base station.

[0104] The present invention is based on the modeling of the downlink feeder link, the uplink feeder link and the ground wireless propagation link. The present invention can obtain the unit content transmission delay τ of the base station respectively. b , the unit content transmission delay τ of LEO satellite s , and the unit content transmission delay τ of the gateway g , and the uplink feeder link power loss P UL and ground wireless propagation link power loss PBS Since the information transmitted between satellites is feature information extracted from historical user request data, the amount of data is very small compared to the content requested by the user, so the delay and energy loss during inter-satellite transmission of information are not considered.

[0105] Subsequently, the present invention calculates the delay reduction gain of the ground base station and the LEO satellite based on the network model and the multi-content sub-library model. The delay reduction gain is the difference between the maximum transmission delay (all content is sent to the user through the core network gateway) and the transmission delay of the edge cache node (base station and satellite) currently used. The specific steps are as follows:

[0106] First, define the function I(x,y) to determine whether the element x is in the vector y. The expression is:

[0107]

[0108] Base Station B i User requests received at time slot t The cache status is From this we can get base station B i The delay reduction gain is:

[0109]

[0110] according to It can be concluded that the satellite S j Collaborative base station collection After the user request is responded to by the service of the base station layer, the unanswered content request is forwarded to the satellite S j ,Right now It can be obtained from formula (11):

[0111]

[0112] LEO satellite cache status is From this we can get the satellite S j The delay reduction gain is:

[0113]

[0114] Then, the satellite S in time slot t can be obtained. j Integration with base station The delay reduction gain is:

[0115]

[0116] The cache update cost mainly considers the energy loss caused by the satellite and base station obtaining new content from the core network. It mainly considers the satellite's uplink feed link and the ground wireless transmission link from the base station to the core network. The orbital height of the LEO satellite is set as the communication distance between the satellite and the core network. The energy loss mainly considers the free space propagation loss, and the expression is:

[0117]

[0118] When the satellite and the base station update their cache, they must do so within a time slot interval t 0 The maximum energy loss is when the current cache state is completely inconsistent with the content in the cache decision, that is, full replacement. Therefore, we use the worst case to calculate the maximum rate at which the core network transmits content to the satellite, and the expression is:

[0119]

[0120] Using Shannon's formula, the uplink feeder link power loss P can be solved as UL for:

[0121]

[0122] From this, we can get the energy consumption of each satellite cache update:

[0123]

[0124] in Represents the amount of new content to be acquired. Similarly, the energy consumed by the base station to acquire new content is:

[0125]

[0126] The cache cost, that is, the energy loss caused by cache updates, is only related to the number of contents updated by the cache node, that is, the difference between the cache state before and after. Since the elements in the cache state are unordered and each content will not be cached repeatedly at the same time, the number of updated contents can be obtained by finding the symmetric difference of the cache state of the two time slots before and after. The expression is:

[0127] D t =|x t Δ x t-1 | (58)

[0128] Then, the LEO satellite S can be obtained respectively. j and ground base station collection The cache cost calculation formula is:

[0129]

[0130] Use P local,t and E local,t Represents the cache cost and utility of the local network to which the current agent belongs:

[0131]

[0132] ξ represents the proportionality coefficient.

[0133] A satellite will collaborate (associate) with several ground base stations over a period of time. Each edge node can be called an agent in the context of reinforcement learning, and several edge nodes form a small network. The local network to which the current agent belongs is the network in which the current agent is located.

[0134] Different resource attributes and performance indicators in the network have different impacts on the system, which are regarded as system benefits and costs. The target system utility function can be expressed as the difference between system benefits and costs to measure the system. Specifically, G t is the traffic revenue obtained by the cache, P t total The present invention aims to maximize the long-term utility of the sub-network, taking the difference between the network delay reduction gain and the network cache update cost as the instantaneous network utility, and obtaining the long-term utility by summing over time.

[0135] (4) Optimization issues

[0136] Finally, the present invention hopes to maximize the long-term utility of STIN, which is generally composed of a low-orbit giant constellation and a ground network. The Walker constellation is a giant constellation composed of polar-orbiting LEO satellites, with an equal number of satellites in each orbit and equidistant distribution. Therefore, the study of caching strategies for LEO satellites in one orbit can be extended to other orbits. To achieve this goal, the present invention constructs a utility optimization problem and considers the cache status of base stations and LEO satellites. As optimization variables:

[0137]

[0138] Where ξ is the proportionality coefficient, and the constraints C1 and C2 are the buffer capacity limits of the satellite.

[0139] 2. Problem Analysis and Solutions

[0140] (1) Problem Analysis

[0141] In scenarios with heterogeneous preferences for different content sub-libraries and a wide range of users, how can LEO satellites better complete content preheating and model preheating when crossing the top and across regions?

[0142] In existing research work, it is often assumed that the content requested by users comes from the same content set, and the satellite can select content from the content set for caching. Based on this assumption, all content requests are within the satellite's line of sight, so the satellite's caching decision can clearly select certain content, but according to real-world scenarios, content that is very popular in one area may be "cold content" in another area. These contents belong to the blind spot of the satellite's caching strategy. Before the cross-region operation, the satellite has not received any relevant content requests, nor does it have any relevant content information. This area is an information island for the satellite. Failure to foresee or include content preferred by users in the new area results in the inability to meet user requests, which reduces the service quality and network utility of STIN. In order to improve the network utility of STIN, the present invention models this scenario into a multi-content sub-library model to facilitate the construction of an optimization problem regarding caching decisions.

[0143] First, since a large amount of data content is generated on the Internet every day, it is difficult to completely fill the blind spots of the satellite's vision with limited computing power and storage scale. Therefore, the present invention focuses on how to make the cache strategy more adaptable to the changing environment. Since the base station serves local users, the user's content preference is relatively stable. The movement speed of LEO satellites is much faster than the rotation speed of the earth. The present invention assumes that when the satellite switches regions, the satellite S j+1 The area covered in the current time period is the same as S j The areas covered in the next time period are basically the same. Therefore, the user request information collected by the two satellites can be used to measure the preference similarity of the two areas, and a satellite-side candidate content index set can be constructed based on the historical request information. This set is defined as a dynamic content set, which only provides the index corresponding to each content, rather than the content itself, in order to make the satellite aware of the existence of the content. When the satellite switches regions, the set will dynamically increase or decrease the content according to the user's request information. During the period of time when the satellite cooperates with the ground base station to transmit content to the user, the content in the service set is considered unchanged. This will fill the blind spot of the cache strategy to a certain extent and reduce the computing overhead.

[0144] Subsequently, after the agent's policy model has been trained for multiple rounds in one scenario, the model is often limited to the specific environment in which it was trained, resulting in poor generalization. It may not be applicable to other tasks or slightly changed environments and needs to be retrained. Therefore, in order to ensure the user experience, LEO satellites should achieve seamless switching, that is, they can quickly adapt to different scenarios.

[0145] (2) MDP Problem Transformation

[0146] To apply reinforcement learning based algorithms, we first reformulate the optimization problem by converting it into an MDP problem.

[0147] Each base station and the satellites that cooperate with it in the satellite-to-ground cache network are considered as an agent, and each agent will make corresponding actions according to the current observed state. In this subnetwork, the environment that each agent can observe is limited. i The observed state is expressed as in Base station B i Receive a content request from the user at time slot t, Base station B i The cache state at time slot t; similarly, the state observed by satellite S can be obtained Assume A t ={a 1 ,a 2 ,...a n} represents the action space of the agent, and each action a i Represents a content cache combination. Each content can be selected or not selected under the constraints.

[0148] The agent performs an action a in each round. i To get the reward R t , based on the objective function proposed in formula (17), the reward function of the agent is designed, and the expression is:

[0149] R t =E agent,t +γ 1 E local,t +γ 2 E total,t (64)

[0150] Since our goal is to improve the service quality of all LEO satellites in orbit and the covered base stations, the reward consists of the agent's local network utility and the global network utility, where γ 1 and γ 2 is the weight parameter, E agent,t is the current utility of the agent, E local,t is the utility of the local network to which the current agent belongs, E total,t This is the utility of all current LEO satellites and coverage base stations.

[0151] (3) Design of collaborative caching strategy algorithm based on meta-reinforcement learning

[0152] The Meta-Reinforcement Learning (Meta-RL) proposed in the present invention can obtain the satellite-ground collaborative caching strategy of STIN, which only requires a few further training steps to achieve fast updates. Compared with the MADDPG (Multi-Agent Deep Deterministic Policy Gradient Algorithm) algorithm, the Multi-Agent Double Delay Deep Deterministic Policy Gradient (Meta-MATD3) algorithm proposed in the present invention enables the agent to obtain the reward of the selected action directly from a variety of task types, i.e., probability distribution, and reduces the situation of overestimation of value. The algorithm proposed in the present invention is more suitable for complex scenarios with multiple content sub-libraries, thereby obtaining higher long-term network utility.

[0153] The meta-learning algorithm proposed in this paper consists of two stages: first, the inner loop, which aims to find a good parameter for a specific task; second, the outer loop, which combines the results of multiple inner loops to find a good initialization state for all tasks. The design of these two loops will be discussed separately below.

[0154] First, the goal of the inner loop is to solve a standard reinforcement learning problem. Here, we regard each cache node (including ground base stations and LEO satellites) as an agent. The regional switching of LEO satellites is based on the distance between the satellite and the ground station. The satellite-base station association matrix is ​​obtained, which is not used as the input information of reinforcement learning. Therefore, the environmental state observed by each agent includes the received user request information and the local cache state, which can be expressed as S t = {r t ,x t The action of the agent is the content combination of the corresponding cell. The size of the action space is related to the number of contents to choose from and the number of contents that can be cached, which can be obtained by calculating the number of combinations. In the meta-training phase, the strategy focuses on the difference between the probability distributions of the agent's execution of tasks, so it can be assumed that there is already an alternative content index library serving the satellite.

[0155] The present invention uses the MATD3 algorithm to train the cache strategy. The algorithm adopts a centralized training and distributed execution architecture. Each agent contains an Actor network π i , two critic networks and And the corresponding target network π i '、 and Through communication between agents, the states, actions, and reward information of the cooperative agents in each round are put into their respective experience replay pools. During the model training phase, the agent randomly samples a batch of data from the experience replay pool and transmits it to the target Actor network π. i 'Get action ai ′, use two target Critic networks to calculate the target Q value:

[0156]

[0157] In order to reduce the deviation of Q value estimation, the minimum value output by the two target networks is used to calculate the target Q value, where γ is the discount factor. Subsequently, the loss function is calculated and the parameters of the two Critic networks are updated. and The loss function expression is:

[0158]

[0159] Where k = 1, 2. Finally, the Actor network is updated by maximizing the Q value of the Critic network. Update the policy parameters with the value of The obtained gradient expression is:

[0160]

[0161] Among them, J(π i ) is the objective function of the Actor network. Finally, the target network π is updated by soft update. i '、 and The parameters are updated.

[0162] The outer loop is designed to combine the results of multiple inner loops and find a good initialization for all these similar tasks. The first step of the outer loop is to select Sample a batch of tasks Tasks are defined as user requests generated based on the popularity of different types of content, simulating the regional popularity differences that the satellite will face. The inner loop uses each task obtained by sampling To train the model. Thus, K experience trajectories for each task can be obtained through the inner loop.

[0163] Denoted as τ j ={(s j,0 ,a j,0 ,r j,0 ,s j,1 ),(s j,1 ,a j,1 ,r j,1 ,s j,2 ),...,(s j,K-1 ,a j,K-1 ,r j,K-1 ,s j,K )}.

[0164] After collecting the experience trajectories, we need to perform cross-task learning based on these trajectories and update the meta-parameters of the agent. and Since the environment in reinforcement learning is often dynamically unknown, the meta-optimization part uses policy gradient to estimate the gradient. Based on MAML, the objective functions of the meta-parameters can be obtained, which are:

[0165]

[0166] Furthermore, the gradient update of the meta-parameters can be obtained, expressed as:

[0167]

[0168] where β a and β c is the update step size. It can be seen that the method in MAML requires accurate calculation of the gradients in the inner and outer loops, which will bring great computational pressure. Therefore, the present invention adopts the improved Reptile (first-order optimization algorithm Reptile) method based on MAML to train the network model. This method can greatly reduce the computational complexity while still achieving the effect of rapid adaptation to the task. The expression is:

[0169]

[0170] where m is is the number of tasks in , and η is the learning rate.

[0171] Since there are multiple agents in the problem considered in this invention, for simplicity, we use θ π Represents the Actor network parameters of all agents, using θ 1 and θ 2 Representing the parameters of the two Critic networks of all agents, the pseudo code of the proposed Meta-MATD3 algorithm is shown below.

[0172] The algorithm flow is as follows:

[0173]

[0174] In satellite communication systems, due to the dynamic nature of satellites, their coverage areas are constantly changing, so it is necessary to count the differences in user preferences in different areas. The base station is responsible for monitoring and recording the request status in each time slot and transmitting this data to the local LEO satellite. The task of the satellite is to record the number of times different contents appear in these requests, recorded as a vector in An element in represents the number of times a corresponding content is requested. The records of different time slots are then weighted and summed, where the fluctuation value close to the current time slot will be given a higher weight to reflect its impact on the current state. Considering that a single satellite may serve multiple base stations, LEO satellites also need to sum the data from different base stations and further calculate their average value. The mathematical expression of this process is as follows:

[0175]

[0176]

[0177] When the satellite needs to switch regions, the satellite in the next region will accumulate the request status. Sent to satellite S j Based on this information, the current satellite will calculate the Manhattan distance between the cumulative request state vectors of the two regions as an indicator to measure the difference in popularity between regions. The expression is:

[0178]

[0179] It reflects the difference in content characteristics between the two regions. The biggest difference is that the user request states of the two regions are completely different. The difference value at this time is recorded as W max .when Greater than υ·W max When υ∈[0,1], the satellite’s actions will tend to explore, and gradually reduce exploration as the reward value increases.

[0180] The location of the ground base station is fixed, and the user group it covers is relatively fixed. Therefore, it can be assumed that the user's content preference is relatively stable, and a fixed local content library is obtained, which contains the content requested by the user during this period. This local content library is composed of one or more content sub-libraries.

[0181] However, LEO satellites will encounter blind spots in the content during movement, which will lead to a decrease in cache hit rate. It is difficult to adjust the policy model quickly in a short period of time, and the content will still be cached according to the previous policy. Next, the present invention proposes a satellite-side alternative content index library, which allows the satellite to continuously have new content to choose from while controlling the size of the action space and keeping the computational overhead within a controllable range. Since the library only stores the index corresponding to each content, rather than the content itself, it does not take up a lot of cache space.

[0182] The data content is unordered and unique, so use a collection Indicates a set of candidate content. The corresponding index stored on the satellite side is used to inform the satellite of the existence of the content. The size is finite, and the upper limit is defined as F alt , the LEO satellite will store the corresponding content indexes of the user requests received in the past in the collection In. Because of the collection The number of stored indexes is much larger than the number of contents cached by LEO satellites. We do not want the update strategy of the collection to bring too much computational overhead, and the user preference changes in the region are not periodic. Therefore, the present invention adopts the First Input First Output (FIFO) strategy algorithm as the collection. The update algorithm of When the size reaches the upper limit, the content index that entered the collection earliest is eliminated first.

[0183] The algorithm flow is as follows:

[0184]

[0185]

[0186] The inventive points of the present invention are: (1) a satellite-base station cache network model is established according to the application background of the satellite-ground fusion cache network, and under this model, a satellite-ground cache strategy optimization problem is constructed for the highly heterogeneous content popularity scenario; (2) a distributed cache strategy for satellite-ground and inter-satellite collaboration based on meta-reinforcement learning is proposed to minimize network service delay and cache update cost.

[0187] Beneficial effects of the present invention: The dynamic cache update method of the present invention fully considers the inter-satellite and satellite-ground collaboration in scenarios with highly heterogeneous content popularity, uses meta-learning to improve the generalization ability of deep reinforcement learning, uses multi-agent reinforcement learning to improve the collaboration of distributed systems, and makes full use of the local environmental information observed by the agent to construct an adaptive threshold (a method for calculating regional content feature differences), effectively improving network energy efficiency.

[0188] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A dynamic cache update method for a satellite-ground fusion network, characterized in that: The following steps are involved: Step 1: Establish a time-varying association matrix between LEO satellite and base station according to the position information of LEO satellite, and build a three-layer cache architecture network model of ground base station, LEO satellite and core network gateway based on the time-varying association matrix between LEO satellite and base station; Step 2: Construct a multi-content sub-library model to characterize the content popularity in different satellite coverage areas; each content sub-library contains multiple contents, representing a certain type of content. Users in different regions will prefer content in different sub-libraries. The content popularity distribution in each region follows the Zipf distribution. There are differences in preferences between different regions, and the content of some sub-libraries is only popular in some regions. Step 3: Build a communication link model between satellites and base stations in the network and establish performance indicators to measure the quality of satellite-to-ground caching strategies; Step 4: First, a satellite-ground and inter-satellite adaptive threshold collaborative caching strategy based on meta-reinforcement learning is proposed to quickly learn the changing characteristics of content popularity between different coverage areas, and then based on the similarity of content features between satellite coverage areas and the set of alternative content indexes at satellite nodes, fast cache updates are achieved during area switching.

2. The dynamic cache update method according to claim 1, characterized in that: In the step 1, it also includes: Step 1: Use a size S J ×B I The matrix G t To describe the association state between the satellite and the base station in each time slot, it is expressed as: in Indicates that the satellite S in time slot t j With base station B i The connection relationship of satellite S j With base station B i In the associated state, 0 is the opposite; Step 2: Deploy caches on the base station side and the satellite side. The cache capacity of each base station is The cache capacity of each satellite is The cache status of the base station is represented as The cache status of the satellite is represented by and Respectively represent base station B i and LEO satellite S j The cth content stored in the cache.

3. The dynamic cache update method according to claim 1, characterized in that: The step 2 is specifically: dividing the content set into M content sub-libraries to represent different types of content. The set is represented as: Users in each region may be interested in the contents in one or more content sub-libraries. There are multiple files in the database, and the popularity ranking of each content will change over time, but the probability of the content in each sub-library being requested by the user always follows the probability P m , expressed as: Content popularity It follows the Zipf distribution, where i is the popularity ranking of the content; In step 2, time is divided into multiple discrete time slots, represented as t=1, 2, ..., T. In each time slot t, users in the area will generate content requests based on content preferences. j The upper limit of the number of user requests received is the sum of the number of users in each coverage area; In step 2, the method of transmitting content to users uses a layered service architecture. First, the local base station responds. If there is no corresponding content, the request is forwarded to the associated satellite. Finally, the remote gateway provides the corresponding content. The base station and the LEO satellite update the cached content through the gateway connected to the remote core network.

4. The dynamic cache update method according to claim 1, characterized in that: The step three comprises: Step 1: Model the downlink feeder link, uplink feeder link and ground wireless transmission link, and obtain the unit content transmission delay τ of the base station respectively b , the unit content transmission delay τ of LEO satellite s , the unit content transmission delay τ of the gateway g , and the uplink feeder link power loss P UL and ground wireless propagation link power loss P BS ; Step 2: Based on the cache architecture network model of step 1 and the multi-content sub-library model of step 2, the delay reduction gain of the ground base station and the LEO satellite is calculated; Step 3: Calculate the symmetric difference of the cache states of the two time slots before and after to obtain the number of updated contents. The expression is: D t =|x t Δx t-1 |(5) Then, we get the LEO satellite S j and ground base station collection The cache cost calculation formula is: Then use P local,t and E local,t Represents the cache cost and utility of the local network to which the current agent belongs: Among them, ξ represents the proportionality coefficient, Indicates the cache status of the base station. Indicates the cache status of the satellite.

5. The dynamic cache update method according to claim 4, characterized in that: The step 2 specifically includes: firstly defining a function I(x,y) for judging whether the element x is in the vector y, the expression is: Base Station B i User requests received at time slot t The cache status is This gives the base station B i The delay reduction gain is: according to Derive with satellite S j Collaborative base station collection After the user request is responded to by the service of the base station layer, the unanswered content request is forwarded to the satellite S j ,Right now From formula 11, we can get: LEO satellite cache status is Thus, we get the satellite S j The delay reduction gain is: Then, we get the satellite S in time slot t j Integration with base station The delay reduction gain is:

6. The dynamic cache update method according to claim 1, characterized in that: In step 4, in the meta-training of meta-reinforcement learning, the outer loop starts from the task set Sample a batch of tasks Tasks are defined as user requests generated based on the popularity changes of different types of content, simulating the popularity differences between regions that the satellite will face. The inner loop uses each task obtained by sampling. To train the model, we obtain K experience trajectories for each task through the inner loop, denoted as τ j ={(s j,0 ,a j,0 ,r j,0 ,s j,1 ),(s j,1 ,a j,1 ,r j,1 ,s j,2 ),...,(s j,K-1 ,a j,K-1 ,r j,K-1 ,s j,K )}, and finally the outer loop performs cross-task learning based on the experience trajectory collected by the inner loop to update the meta-parameters of the agent and 7. The dynamic cache update method according to claim 1, characterized in that: In step 4, when the satellite needs to switch regions, the satellite in the next region will accumulate the request status Sent to satellite S j , the current satellite will calculate the Manhattan distance between the cumulative request state vectors of the two regions as a measure of the difference in content characteristics between the regions. The expression is: It reflects the difference in content characteristics between the two regions. The biggest difference is that the user request states of the two regions are completely different. The difference value at this time is recorded as W max ,when When υ∈[0,1], υ is a proportional parameter, the satellite's action tends to explore, and as the reward value increases, exploration is reduced. f represents the user's request, f represents the content, The meaning is that the satellite S j The number of requests for different contents received by the associated base station from users during the period from t0 to t is represented by a vector.

8. The dynamic cache update method according to claim 1, characterized in that: In step 4, use the collection Indicates a set of candidate content. The corresponding index is stored on the satellite side. The upper limit of the size is defined as F alt , the LEO satellite stores the corresponding content indexes of the user requests received in the past in the collection Then the first-in-first-out algorithm is used as the collection The update algorithm of When the size reaches the upper limit, the content index that entered the collection earliest is eliminated first.

9. The dynamic cache update method according to claim 6, characterized in that: In the inner loop of meta-training, the MATD3 algorithm is used to perform small sample learning on different distributions. The specific steps are as follows: Step S1: Initialize meta-strategy parameters and critic network parameters θ π ,θ 1 and θ 2 , discount factor γ, outer learning rate η, inner learning rate ε, task distribution Step S2: Each task sampled using the outer loop Train the model; Step S3: Using strategy parameters Collecting experience trajectories τ j ; Step S4: Calculate the reward R on the experience trajectory using formula (30), and calculate the critic network loss function according to formula (32); Step S5: Through stochastic gradient descent Update the parameter θ 1 and θ 2 , ε represents the inner learning rate, represents the loss function, Represents the network parameters in the inner loop; Step S6: According to formula (33) and Calculate the gradient to update the parameter θ π , represents the policy parameters, represents the gradient, ε represents the inner learning rate; Step S7: Soft update the target network parameters, where k = 1, 2, is the meta-parameter of the agent, are the parameters of the target network.

10. The dynamic cache update method according to claim 6, characterized in that: The improved Reptile method based on meta-learning is used to train the network model, and the expression is: where m is is the number of tasks in the network, η is the outer learning rate, a is the actor network, and c is the critic network.

Images