Space-air-ground integrated internet intelligent edge cache method

Abstract

The application discloses a space-air-ground integrated Internet cache content updating method, and belongs to the field of satellite communication. A cache content updating model is established under the space-air-ground integrated Internet scenario with relay nodes having cache capability. The cache decision of each relay node to satellite broadcast content is modeled as a Markov decision process. The relay nodes are AP clustered according to the characteristics of the relay nodes. The cache decision of each relay node is optimized by using imitation learning. The cluster head node of the relay node cluster periodically reports the cache strategy neural network parameters to the satellite, and the cache strategy neural network parameters are summarized and integrated into integrated strategy parameters and distributed to each relay node. Until the cache strategy of the relay node converges. The imitation learning method in deep reinforcement learning is used to learn the change rule of the dynamic change of content popularity, so that the cache hit rate is improved. On the other hand, a low-communication-traffic federated learning mechanism based on AP clustering is provided, so that the learning efficiency of the cache strategy of each relay node is improved at the cost of low communication overhead.

Description

Technical Field

[0001] This invention belongs to the field of satellite communication and relates to an intelligent edge caching method for an integrated air-space-ground internet based on low-traffic federated imitation learning. Background Technology

[0002] With the rise of short video applications, wireless networks are evolving from traditional text and voice services to content-centric services such as streaming media. Because satellites can provide wide-area coverage, future space-based information networks that integrate space and ground capabilities can be widely used for content delivery services. Many internet companies, including Google and Facebook, have launched or plan to launch internet access satellites to provide content access services to users in remote areas. This necessitates a shift in satellite network architecture from traditional connection-oriented architectures to information-centric networks. Information-centric networks have two main characteristics: name-based routing and network caching. The former means that specific content can be retrieved from the network by its name without requiring a host identifier, while the latter indicates that network elements (such as routers) are equipped with a certain amount of memory and can cache data. In terrestrial networks such as the internet, network caching has proven to be an effective method for improving network throughput and reducing latency.

[0003] Drawing inspiration from edge caching, in an integrated space-air-ground network, the caching capabilities of airborne and ground relay nodes can be leveraged to cache frequently accessed content, allowing user requests for this content to be delivered directly through the relay nodes. This mechanism offers two main advantages: firstly, it effectively reduces transmission latency, thereby improving the quality of service for users; secondly, it reduces the load on satellite network segments, utilizing limited satellite communication resources to meet the growing business demands of users. Current research in space-based content delivery networks primarily focuses on cache placement strategies and cached content distribution mechanisms.

[0004] The cache content placement strategy is crucial in determining the cache hit rate, and its main function is to select which files to cache on the caching device. An efficient content placement strategy is reflected in a high cache hit rate. Other network performance metrics, such as average content access latency, throughput, energy utilization, and satellite bandwidth consumption, are also used as optimization targets for cache placement strategies. Currently, most edge cache content update strategies in satellite networks are based on optimization or metaheuristic methods, making it difficult to cope with dynamic changes in content popularity and user preferences.

[0005] (1) Existing reinforcement learning-based caching strategies can learn changes in content popularity from environmental feedback to optimize caching decisions. However, the determination of immediate rewards in mainstream methods cannot be obtained immediately after the current caching action, but requires waiting for the system to run for a period of time. This results in a mixture of immediate rewards between multiple consecutive actions, and unclear and inaccurate immediate rewards can mislead the agent, leading to low convergence efficiency. Extensive interactive exploration during training will cause resource losses to the satellite internet system.

[0006] (2) To improve the convergence efficiency of each relay node and reduce the time required for training agents through interaction with the environment, existing methods utilize federated learning frameworks to periodically summarize and distribute the cached decision model parameters of each relay node, thereby improving the learning efficiency of caching policies. However, this mechanism results in a large communication overhead for all agents to transmit policy network parameters during the summary model parameter stage. Summary of the Invention

[0007] This invention targets integrated air-space-ground internet scenarios where relay nodes have caching capabilities. It utilizes imitation learning methods in deep reinforcement learning to learn the dynamic changes in content popularity, thereby improving cache hit rate. On the other hand, it proposes a low-communication-volume federated learning mechanism based on AP clustering, which improves the learning efficiency of caching strategies of each relay node at the cost of lower communication overhead.

[0008] A method for intelligent edge caching of the air-space-ground integrated Internet includes the following steps:

[0009] Step 1: Establish a cached content update model for an integrated air-space-ground internet scenario where relay nodes have caching capabilities;

[0010] Step 2: Model the caching decision for satellite broadcast content at each relay node as a Markov decision process;

[0011] Step 3: Perform AP clustering on relay nodes based on their characteristics;

[0012] Step 4: Each relay node optimizes its caching decisions using imitation learning;

[0013] Step 5: Instruct the cluster head node of the relay node cluster to periodically report the caching strategy neural network parameters to the satellite, summarize and integrate them into an integrated strategy parameter, and distribute it to each relay node;

[0014] Step Six: Repeat Steps Four and Five until the relay node caching strategy converges.

[0015] The present invention adopts the above technical solution and has the following advantages compared with the prior art:

[0016] 1. The imitation learning method solves the problem that the immediate benefits of caching content placement are difficult to design and are easily affected by other actions, thus improving the optimization effect of the caching decision algorithm;

[0017] 2. To address the issue of high communication costs in caching strategies based on federated reinforcement learning, the clustering results of relay nodes are learned from their features through the AP clustering process. The cluster head node of each relay group then reports the caching strategy model parameters on behalf of all its member nodes, thereby reducing the amount of communication required by the federated learning framework.

[0018] 3. By designing a Q-value neural network using an attention mechanism, the content of the current satellite broadcast is fully compared with the content in the relay node cache, which is more conducive to selecting the appropriate content for replacement. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of an integrated air-space-ground internet scenario with relay caching capabilities according to the present invention;

[0020] Figure 2 This is a schematic diagram illustrating the service process of whether the edge node matches the content.

[0021] Figure 3 This is a diagram of the neural network structure for the imitation learning strategy of this invention;

[0022] Figure 4 This is a schematic diagram comparing the clustering imitation learning mechanism of this invention with other similar methods. Detailed Implementation

[0023] A method for intelligent edge caching of the air-space-ground integrated Internet includes the following steps:

[0024] Step 1: Establish a cached content update model for an integrated air-space-ground internet scenario where relay nodes have caching capabilities;

[0025] Step 2: Model the caching decision for satellite broadcast content at each relay node as a Markov decision process;

[0026] Step 3: Perform AP clustering on relay nodes based on their characteristics;

[0027] Step 4: Each relay node optimizes its caching decisions using imitation learning;

[0028] Step 5: Instruct the cluster head node of the relay node cluster to periodically report the caching strategy neural network parameters to the satellite, summarize and integrate them into an integrated strategy parameter, and distribute it to each relay node;

[0029] Step Six: Repeat Steps Four and Five until the relay node caching strategy converges.

[0030] like Figure 1 As shown, this invention focuses on the downlink of an integrated air-space-ground internet scenario with relay caching capabilities (content-centric). It assumes that users cannot directly communicate with satellites; content from internet servers is relayed by LEO satellites to drones or ground stations before being transmitted to users. The system operation can be discretized into a time series containing equal time slices.

[0031] Step 1: Establish a cached content update model for an integrated air-space-ground internet scenario where relay nodes have caching capabilities;

[0032] N U One drone and N T The set of ground relays is represented as Where L = N U +N T Each is equipped with a cache unit of limited capacity. A directory containing all N contents is represented as... For ease of discussion, let's assume all content has the same data size, S. F Each relay's cache unit has an M×S F A 1-bit memory can cache up to M items. K users are distributed across the network coverage area, using... This indicates that the user is assumed to be stationary.

[0033] In an integrated air-space-ground network scenario, data requests from ground user terminals can be satisfied in the following two ways: Figure 2 As shown:

[0034] (1) Figure 2 (a) represents the ideal scenario: when the requested content is cached locally on the connected relay, the local relay directly transfers the file to the user. Even so, the request will still be recorded for subsequent relay node cache updates.

[0035] (2) Figure 2 (b) indicates that when the requested content is not cached in the local relay node's cache, the LEO constellation retrieves the content from the cloud data center. Specifically, each relay node counts the content requests not cached locally within each time slice and periodically reports the statistics to the satellite. For the aforementioned missed content, the satellite backhaul link retrieves the content from the cloud data center and broadcasts these files to the relay nodes. The relay nodes then distribute their decisions regarding whether to cache the currently broadcast content.

[0036] This satellite-broadcast-based content distribution mechanism leverages the wide-area coverage of satellites to implement a combined active and passive caching mechanism, which helps improve cache update efficiency. For example, if a user's content request f0 fails at relay node R0, the satellite retrieves f0 from the cloud data center and broadcasts it to all relay nodes. All relay nodes receive content f0 and determine whether to cache it; therefore, relay R1 can pre-cache this content before a local user requests f0.

[0037] Assume that a user requests at most one piece of content at any given time, let vector Indicates user u k The content request at time t, i.e.:

[0038]

[0039] It is generally assumed that the request probability of each piece of content follows a Zipf distribution, and its popularity changes over time. If at time t, content f... n Ranked rth among all content based on access frequency n If there are 1, then the average probability of being requested can be approximately expressed as:

[0040]

[0041] in γ is a decay constant describing the distribution of content popularity. The larger the γ is, the more uneven the distribution of file popularity.

[0042] Let matrix C represent the cache placement results of each relay node, i.e.

[0043]

[0044] Therefore, user u k Is the file being accessed in the cache of its access relay node? Representation, where vector Indicates user u k The results of the access relay selection.

[0045] like Then user u k The requested data is cached in the relay node and can be directly transmitted through the relay node; if This requires accessing the cloud server via satellite to obtain the content, and then forwarding the content to the user terminal through a relay node.

[0046] At time t, the system's content transmission service volume to users is the sum of the content data requested by each user through the system.

[0047]

[0048] The satellite needs to transmit content that is not cached by the local relay node. The amount of data to be transmitted at time t is:

[0049]

[0050] Where S F For the amount of data in a single piece of content, Indicates user u k The content requested at time t is in the relay of the user access, so it does not need to be obtained from the cloud data center using the space-based segment.

[0051] Therefore, the cache hit rate of the integrated air-space-ground internet can be expressed as:

[0052]

[0053] In summary, the optimization model corresponding to the relay selection and caching strategy problem of this invention can be expressed as follows:

[0054] opt.

[0055] st

[0056]

[0057] Where the optimization variable C t Let t represent the cache decision matrix of each relay node at time t. The optimization objective is to maximize the long-term cumulative cache hit rate. Among the constraints, the first constraint means that a user can request at most one piece of content in a time slice and a user can only access one relay node to make a data request at a certain time. The second constraint means that the cache space of the relay node is limited.

[0058] Because the popularity of content and user preferences in real-world systems change over time, caching decisions must address this dynamic environment. Existing methods primarily focus on two approaches to address this issue:

[0059] (1) Predict popularity and make decisions based on the predicted content popularity;

[0060] (2) By using reinforcement learning and leveraging the agent's ability to explore the environment, the agent can explore and learn the changing patterns of popularity through interaction with the environment.

[0061] This invention adopts the second approach, which utilizes an intelligent agent to explore the benefits of various caching strategies to a certain extent and track the dynamic popularity of content. Its advantage lies in avoiding the computational overhead required to continuously predict the popularity of massive amounts of content in a real system.

[0062] In step two, each relay node is treated as an agent, and its sequence cache update decision is modeled as a Markov decision process. The specific method is as follows:

[0063] As the system operates, satellites continuously retrieve missed content from the cloud data center. Each relay node must constantly decide whether to update its cached content; that is, whether to add the currently broadcast content to its cache, and if so, decide which cached content to replace. Since the state of the content in the relay node's cache at a later time step depends only on the current cached content state and the agent's caching actions, caching decisions can be modeled as a Markov decision process. The core elements of a Markov decision process are the environment observed by the agent, the actions performed, and the benefits derived from those actions. These three elements will be discussed in detail below.

[0064] 1. State Space

[0065] To make caching decisions, relay nodes need to observe the following three main types of information:

[0066] (1) Content that has been cached Information

[0067] (2) The content currently being broadcast by satellite f B Information

[0068] (3) Historical request content of this relay node Information, where ΔT is the time interval, k T Let f be the length of the backtracking window, where element f iΛt This represents the mean of all request content features within the interval (t-iΔt, t-(i-1)Δt), where i indicates that there are multiple time windows, i ∈ {0, 1, ..., k}. T} Each f iΛt It is the average of all request content features within each time window, so the average features of each time window are spliced ​​together to form the historical content request feature Rq containing time sequence information.

[0069] For relay nodes, content f n Feature information s fn This includes its local time-series popularity and the interest of local connected users in it.

[0070] Content f n The local popularity of content is explicitly reflected in the number of requests, which can be calculated. To fully perceive content f n The present invention statistically analyzes the historical popularity sequence with an interval of ΔT and a sequence length of k to predict the trend of popularity. T Therefore, the popularity time series information can be represented as in (tk)T Δt,t-(k T Within the time interval -1)Δt), f n Local popularity of relay nodes.

[0071] In summary, content f n The feature is defined as: Relay node R l The state information observed by the corresponding agent is an integration of three types of content information, namely

[0072] 2. Movement space

[0073] Relay node R l You can either replace a selected cached content with the content currently requested, or leave the cached state unchanged. Therefore, for the content f currently being broadcast by the satellite... B The set of all possible actions that this agent can take is a∈{0,1,2,…,M}, where a=0 indicates that the relay node does not cache content f. n a = m means that the agent selects to remove the original cached content at position m and sets the content f to f. B Place it at position m.

[0074] Action a l Based on the one-hot encoding, the vector ha = [ha0, ..., ha] is composed of 0s and 1s. m ,…,ha M ], If ha m If = 1, then a = m.

[0075] However, due to the massive amount of data in real-world systems and the numerous cache locations, to avoid the curse of dimensionality caused by an excessively large M in the DRL, which would hinder training convergence, the content at each cache location... The actions are sorted based on a weighted index of local popularity and interest. The M′ items with the lowest ranking index are used as replacement candidates (M′ << M), thus reducing the action space to a ∈ {0, 1, 2, ..., M′}. This mechanism combines popularity and interest indices, simply and directly solving the problem of excessively large action spaces leading to decreased agent convergence in the action caching problem.

[0076] 3. Instant earnings

[0077] The immediate benefit of a caching strategy should demonstrate whether it hits the requested content. The goal of optimizing a caching strategy is to maximize the number of hits, i.e.

[0078] Step 4: Each relay node optimizes its caching decisions using imitation learning. The specific method is as follows:

[0079] Imitation learning, as a type of reinforcement learning method, is suitable for solving sequential decision-making problems where immediate benefits are difficult to define, but expert decisions can be imitated. During the training phase, the Belady optimal caching policy, with a known future request sequence, is used as the expert policy. The agent imitates the decisions of this ideal expert policy, learning the mapping from environmental states to optimal actions. Thus, during the testing phase, it can make caching decisions based on observations of the environment even when future access sequences are unknown.

[0080] 1. Expert Strategy

[0081] Knowing the future access sequence, experts can assess the impact of each action on the cache hit rate and thus select the optimal action. This invention chooses the Belady strategy as the expert strategy, determining whether the current content should be cached based on the distance to the next request and the sequence number of the content to be removed. The distance to the next request of the content is d. t (f n The cached file is defined as the number of requests between its next request and the current request. For M′ cached files... With the current broadcast content f B set The Belady strategy chooses not to cache the content with the largest distance in the next request.

[0082] Belady's expert actions in strategy selection are:

[0083]

[0084] In many imitation learning tasks, querying experts is expensive and limited; however, in the cache placement problem of this invention, since the historical access data of users is known during the training phase, the computational overhead of calculating expert actions is very low, and thus a large number of corresponding expert decisions for a large number of states can be obtained almost without limitation, resulting in sufficient training samples.

[0085] 2. Training methods

[0086] Once the state sequence is collected That is, update the policy π(θ) to better approximate the expert policy π. * The update of π(θ) is achieved through the loss function. Implementation. This invention employs a log-likelihood-based behavioral cloning loss to enable the learning policy neural network to select π with the highest probability. * The best action to choose * ,Right now:

[0087]

[0088] Among them, ha mand The one-hot vector *ha* of the agent's actions and the one-hot vector *ha* of the expert's actions are respectively. * The m-th component represents the content cached at the m-th position that has been removed.

[0089] To enhance supervision during training and improve convergence efficiency, this invention adds a regression of the next request distance as an auxiliary task to the aforementioned behavior cloning loss. Borrowing from the architecture of multi-task neural networks, a branch is added to the policy network. This branch outputs an estimate of the next request distance, which is then trained using mean squared error.

[0090]

[0091] in The logarithm of the distance is used again to estimate the content of the neural network.

[0092] The two losses mentioned above are weighted and used as the overall loss for training the policy network, i.e. 3. Policy Neural Network

[0093] Imitation learning policy neural networks employ attention mechanisms, such as... Figure 3 As shown. Figure 3 The medium-policy neural network will store information from various contents within the cache. Treating it as multiple queries, the historical request information extracted by LSTM is used. and current request information As a key, an attention mechanism is used to compare multiple queries with the key, thus focusing on learning the relationship between each cached content in the query and the key. This facilitates the subsequent selection of the least advantageous content already in the cache based on the time-series request information in the key, and the removal of such content from the cache to achieve cache updates.

[0094] 4. Algorithm Flow

[0095] Behavioral cloning is a basic imitation learning method that uses the expert's decision-making (state, action) pairs as the input and output of a neural network to train the mapping between input states and output actions, thereby fitting the expert's behavior.

[0096] However, this simple and direct supervised learning approach faces the following problems in continuous decision-making problems:

[0097] (1) The learner’s actions will affect the future input state during the execution of the learning strategy, which violates the key assumption of supervised learning that all training samples are independent and identically distributed.

[0098] (2) Error Accumulation. Intuitively, during the testing phase of the algorithm's actual use, since the agent learns an approximation of the expert's optimal policy, it will eventually make mistakes and choose suboptimal actions. Once an error is made, it will face cached states that did not appear during the training phase, because the training state data during the training phase all come from the sequential decisions of the expert policy. The agent cannot make effective decisions when faced with cached states that did not appear during the training phase, and the state at the next moment is very likely still not within the sample coverage of the training process, thus leading to the compound accumulation of errors.

[0099] To address the aforementioned issues, this invention utilizes the DAgger algorithm, which embodies the concept of online learning, and proposes an imitation learning-based cache placement algorithm (IL-CP), as shown in Algorithm 1.

[0100]

[0101]

[0102] IL-CP follows the currently learned policy π(θ) instead of the expert policy π during training. * Collect state sequences This forces the distribution of states during training to match the distribution of states during testing, thus avoiding compound errors. Furthermore, as the IL-CP updates the policy, the current policy π(θ′) is compared with the previously collected... The strategies π(θ) become increasingly different, leading to changes in the training state distribution. The test state distribution deviates from the expected distribution. To mitigate this, the data is re-collected under the current policy π(θ′). Every T U = 5000 parameter updates, i.e., periodically updating the state sequence This method directly and simply solves the problem of inconsistency between training and testing states, thereby limiting the difference between the training process and the actual data distribution to a certain range.

[0103] In step five, the cluster head node of the relay node cluster periodically reports the cached strategy neural network parameters to the satellite, which are then aggregated and integrated into integrated strategy parameters and distributed to each relay node. The specific method is as follows:

[0104] The IL-CP method, which allows each relay node to train independently, has the advantage of fully distributed decision-making without the need for additional communication. However, this also means that each relay node agent needs to spend a lot of time exploring the environment, collecting experience, and training during its training process. In real-world applications, this training process can cause the system to malfunction and waste system resources. Therefore, it is necessary to reduce the time required for the algorithm to explore and collect experience.

[0105] To address this issue, existing methods propose an edge caching strategy based on federated reinforcement learning. Each node reports the neural network parameters of the caching strategy and broadcasts the integrated parameters to all nodes, thereby enabling training experience sharing among caching nodes, effectively improving training convergence efficiency and reducing the number of interactions with the system environment. However, this mechanism comes at the cost of significant communication overhead for node parameter reporting. Compared to terrestrial networks, space-based information networks have scarcer spectrum and energy resources, thus requiring greater attention to compressing communication overhead.

[0106] like Figure 4 As shown, DQN-CP represents a general cache placement strategy based on the Deep Q Network (DQN) method, IL-CP represents a cache replacement strategy based on general imitation learning (IL), and LCIL-CP represents the edge caching method based on low-traffic federated imitation learning proposed in this invention. Figure 4 The simulation presents the daily cache hit rate changes for each method over a one-month period, observed through observation. Figure 4 It can be observed that the method proposed in this paper has significant advantages over the general DQN method, and improves the average monthly cache hit rate by 13.83% compared with the ordinary imitation learning method.

[0107] To improve training efficiency and reduce communication overhead caused by the federated learning framework, this invention proposes a low-communication-volume federated imitation learning algorithm for relay node caching strategies. The main ideas are reflected in the following two aspects:

[0108] (1) Leveraging the advantages of federated learning, relay nodes can share models to improve the learning efficiency of caching strategies.

[0109] (2) The AP clustering algorithm is used to reduce the additional communication overhead of federated learning, thereby improving the availability of the algorithm.

[0110] This invention proposes a clustering method to group relay nodes into clusters. Since relay nodes within a cluster have high similarity, their caching models are also quite similar. There is no need for redundant reporting of all node parameters; instead, the core node only needs to represent other cluster nodes to send the caching strategy parameters to the satellite for model aggregation. Therefore, this invention proposes a low-traffic federated imitation learning caching strategy (LCFL-IL-CP).

[0111] The main reason for choosing the AP clustering method is that it does not require manually specifying the number of clusters, but rather adaptively clusters according to the distribution of sample data and obtains the cluster center node for each class.

[0112] During algorithm training, the total number of training iterations is divided into T steps. FThe overall process for each round is shown in Algorithm 3.

[0113]

[0114]

[0115] Deep reinforcement learning, as a sequential decision-making method that does not rely on precise pre-modeling, has been introduced into edge caching strategies for satellite networks. Leveraging the agent's ability to explore the environment, deep reinforcement learning tracks trends in popularity through interaction with the environment, thereby optimizing cache update strategies and improving cache hit rates.

[0116] This invention primarily targets integrated air-space-ground internet scenarios where relay nodes possess caching capabilities. It utilizes imitation learning methods in deep reinforcement learning to learn the dynamic patterns of content popularity changes, thereby improving cache hit rates. On the other hand, it proposes a low-communication-volume federated learning mechanism based on AP clustering, which improves the learning efficiency of caching strategies for each relay node at the cost of lower communication overhead.

Claims

1. A space-air-ground integrated intelligent edge caching method for the Internet, characterized in that... Includes the following steps: Step 1: Establish a cached content update model for an integrated air-space-ground internet scenario where relay nodes have caching capabilities; Step 2: Model the caching decision for satellite broadcast content at each relay node as a Markov decision process; Step 3: Perform AP clustering on relay nodes based on their characteristics; Step 4: Each relay node optimizes its caching decisions using imitation learning; Step 5: Instruct the cluster head node of the relay node cluster to periodically report the caching strategy neural network parameters to the satellite, summarize and integrate them into an integrated strategy parameter, and distribute it to each relay node; Step Six: Repeat Steps Four and Five until the relay node caching strategy converges.

2. The integrated air-space-ground intelligent edge caching method for the Internet according to claim 1, characterized in that... Step one establishes a cache content update model for an integrated air-space-ground internet scenario with relay nodes having caching capabilities. The specific process is as follows: N U One drone and N T Each ground relay has buffering capabilities, and their set is represented as follows: Where L = N U +N T A directory containing all N items is represented as Assume all content has the same data volume, S. F Each relay's cache unit has an M×S F A 1-bit memory can cache up to M items; K users are distributed across the network coverage area, using... express; The optimization problem of the established cache update strategy is: in Indicates user u k For directory f n The request situation, Representing user u k At this moment, the directory f is requested. n C represents the cache placement result matrix of each relay node. t C represents the cache decision matrix of each relay node at time t. t The matrix element c l,n Represents relay node R l For directory f n The cached result, c l,n =1 means relay node R l The cache contains directory f n The optimization objective represents maximizing the long-term cumulative cache hit rate, CH t For the cache hit rate of the integrated air-space-ground internet; The first constraint states that a user can request a maximum of one piece of content per time slice and can only access one relay node to request data at any given time; the second constraint states that the relay node has limited cache space.

3. The integrated air-space-ground intelligent edge caching method for the Internet according to claim 2, characterized in that... In step two, the caching decision for satellite broadcast content at each relay node is modeled as a Markov decision process, specifically as follows: 2.1 The environmental state that an intelligent agent needs to observe for decision-making mainly includes the following three types of information: (1) Content that has been cached Information; (2) The content currently being broadcast by satellite f B Information; (3) Historical request content of this relay node Information, where ΔT is the time interval, k T Let f be the length of the backtracking window, where element f is the length of the backtracking window. iΛt This represents the mean of all request content features within the interval (t-iΔt, t-(i-1)Δt); 2.2 The action space of the agent is as follows: Relay node R l You can only replace a selected cached content with the content currently requested, or leave the cached state unchanged; therefore, for the content f currently being broadcast by the satellite... B The set of all possible actions that this agent can take is a∈{0,1,2,...,M}, where a=0 indicates that the relay node does not cache content f. n a = m means that the agent selects to remove the original cached content at position m and sets the content f to f. B Place it at position m; Action a l Based on the one-hot encoding, the vector ha = [ha0, ..., ha] is composed of 0s and 1s. m ,…,ha M ], If ha m =1, then a = m; 2.3 The immediate benefit of an agent should represent the number of times its cache is hit, i.e. The goal of optimizing a caching strategy is to maximize the number of hits, i.e.

4. The integrated air-space-ground intelligent edge caching method for the Internet according to claim 3, characterized in that... In step four, each relay node optimizes its caching decisions using imitation learning. The specific process is as follows: 4.1 Collect state sequences at fixed time intervals. 4.2 In the state sequence A batch of sequence decision states were sampled. t represents time, l represents the midpoint of the sampling time, and H represents the sampling window size; 4.3 Given the sequence of future requests, select an expert decision based on the Belady cache replacement strategy; 4.4 Obtaining each decision state using the agent's cache replacement strategy π(θ) Caching decisions; 4.5 Calculate the overall loss for this state sequence. 4.6 Based on the overall loss Update the parameters θ of the policy network π(θ) using gradient descent.

5. The integrated air-space-ground intelligent edge caching method for the Internet according to claim 4, characterized in that... In step five, the cluster head node of the relay node cluster periodically reports the caching strategy neural network parameters to the satellite, which are then aggregated and integrated into integrated strategy parameters and distributed to each relay node. The specific process is as follows: 5.1 Each relay node R l Following step four, the caching strategy is optimized using imitation learning, and the neural network parameters θ of the caching strategy are updated. 5.2 Every Δt F The next update involves the neural network parameters of the cache strategy, and the cluster head node R of each relay node cluster obtained through clustering in step three. l ∈{R G1 ,…R GP Report the caching strategy parameters of each relay node cluster to the satellite. 5.3 After the satellite aggregates the caching strategy parameters of the cluster head nodes of each relay node cluster, it calculates the integrated parameters by weighted averaging the collected parameters. 5.4 The satellite will integrate parameters Broadcast to all relay nodes; 5.5 All relay nodes update their local policy parameters using integrated parameters. Then proceed to step four to optimize the parameters of each policy using imitation learning.

Images