A Low-Earth Orbit Constellation Cache Placement Method and System Incorporating Edge Computing

Abstract

The present invention discloses a method and system for cache placement in a low-Earth orbit constellation integrated with edge computing. The environmental state of a satellite is input into the private model of the satellite, and the private model of the satellite outputs the cache placement decision of the satellite. The process of obtaining the private model of the satellites in the low-Earth orbit constellation includes: the satellite obtains the first model of the satellite through distributed reinforcement learning training, and the satellite uploads the first model to the ground control center; the ground control center regularly fuses the received first models based on federated learning to obtain a shared model, and the ground control center distributes the shared model to each satellite as the private model of the satellite. This method solves the problem that previous cache placement cannot make intelligent caching and cooperation decisions well for complex environments such as low-Earth orbit constellations, which is beneficial to reducing latency, alleviating link congestion, improving the adaptability of agents to the environment, and protecting the privacy and security of node shared data.

Description

Technical Field

[0001] The present invention relates to the technical field of edge computing collaborative caching, and particularly to a caching placement method and system for low-earth orbit constellations integrating edge computing. Background Art

[0002] In recent years, commercial spaceflight has developed rapidly, and low-earth orbit constellation construction plans aiming to serve the global population have been carried out in full swing. However, when a large number of users access the Internet through satellites to obtain services, in addition to long delays, it is also easy to cause link congestion. Edge computing can well solve the above problems. With the increasing density of constellation construction and the continuous improvement of the on-board payload data processing and storage capabilities, integrating low-earth orbit constellations that are convenient for computing has become a feasible solution. It is predicted that mobile video streams will account for 72% of the entire mobile data traffic in the future. A unique attribute of such services is that the content is highly concentrated, and some popular content will be repeatedly requested asynchronously. Therefore, caching the same content in low-earth orbit satellite nodes has significant advantages in reducing service acquisition latency and alleviating the heavy overhead of network backhaul.

[0003] Caching placement is crucial for improving the user service experience. The key to caching placement lies in how to make different caching placement decisions according to the service requirements of different user groups, and consider constraints such as cost, capacity, and latency. Although relevant research has proposed many pioneering works, such as caching placement based on convex optimization, graph theory, or heuristic methods, which have achieved good results. However, due to the complexity and dynamics of the low-earth orbit constellation space communication environment, traditional optimization methods are difficult to provide real-time service performance guarantees. Therefore, leveraging AI technology will become a major trend in solving the caching placement problem. However, current related research still lacks consideration of some issues. For example, how should the agent be trained; how should the agent achieve better self-update; privacy protection of training data, etc. Therefore, it is very important to propose a caching placement mechanism for low-earth orbit constellations integrating edge computing. Summary of the Invention

[0004] The present invention aims to at least solve the technical problems existing in the prior art, and particularly innovatively proposes a caching placement method and system for low-earth orbit constellations integrating edge computing.

[0005] To achieve the above object of the present invention, according to the first aspect of the present invention, the present invention provides a cache placement method for a low-earth orbit constellation integrating edge computing, which obtains the environmental state of a satellite in real time and inputs the environmental state of the satellite into the private model of the satellite, and the private model of the satellite outputs the cache placement decision of the satellite; the cache placement decision includes the decision result of whether the satellite places a cache and the ratio of the satellite requesting cooperation from satellites other than the satellite in the low-earth orbit constellation; the environmental state includes the number of various types of requests faced by the satellite, the cache placement status of satellites adjacent to the satellite, and the remaining processing capacity of the satellite; the process of obtaining the private model of the satellite in the low-earth orbit constellation includes: the satellite obtains the first model of the satellite based on distributed reinforcement learning training, and the satellite uploads the first model to the ground control center; the ground control center fuses the received first models regularly based on federated learning to obtain a shared model, and the ground control center distributes the shared model to each satellite as the private model of the satellite.

[0006] The above technical solution: This method establishes a cache placement mechanism based on federated reinforcement learning for a low-earth orbit constellation integrating edge computing. This method regards each low-earth orbit satellite node as an independent agent, makes decisions quickly based on its own environmental state information using its own private model, and endows each low-earth orbit satellite node with intelligent decision-making ability through distributed reinforcement learning to judge whether to place a certain type of cache content or request cooperation from other nodes, reducing transmission delay and link congestion, and effectively improving the ability of low-earth orbit satellite payloads to accurately make cache decisions in complex space environments. At the same time, by uploading the first model of the agent obtained through its own training to the ground control center for sharing, fusing the model through federated learning, and then distributing it to the low-earth orbit satellite nodes for retraining, the purpose of continuously improving the model and continuously learning the decision-making ability of other low-earth orbit satellite nodes is achieved, effectively improving the training speed and decision-making accuracy of low-earth orbit satellite nodes. The cache placement of this method is dynamically changing. For a complex low-earth orbit constellation, only dynamic cache placement can effectively achieve the purpose of improving service quality and reducing costs, and the dynamic policy is also optimal for resource utilization. This method is conducive to solving the problem that previous cache placement cannot make intelligent cache and cooperation decisions well for complex environments such as low-earth orbit constellations, is conducive to reducing delay, alleviating link congestion, enhancing the adaptability of agents to the environment, and protecting the privacy and security of node shared data.

[0007] In a preferred embodiment of the present invention, the satellite obtains the first model of the satellite based on distributed reinforcement learning training, specifically including: Step S1, constructing the neural network, decision space A, environmental state space S and reward function R of the satellite t ; The Among them, X(t) represents the output value of the cache placement decision model of the satellite at time t; Cs Represents the cost of cached content; Represents the proportion of the cached content s at time t that requires the cooperation of the neighboring satellite set v of the satellite in the LEO constellation for caching; Type represents the number of various types of requests faced by the satellite; Represents the cached content in the inter-satellite link e i The average transmission delay; Represents the inter-satellite link e i Is a judgment function on whether it is the path between the satellite and the cooperative satellite. When the inter-satellite link e i Is the path between the satellite and the cooperative satellite, Is equal to 1. When the inter-satellite link e i Is not the path between the satellite and the cooperative satellite, Is equal to 0. E represents the set of inter-satellite links in the LEO constellation; Penalty represents the penalty term for violating the constraint; Step S2, based on the ε-greedy policy for memory accumulation, stores the memory in the experience replay space. The memory is the vector (S t , A t , R t , S t+1 ), where S t Represents the environmental state of the satellite at time t, S t ∈S; S t+1 Represents the environmental state of the satellite at time t + 1, S t+1 ∈S; A t Represents including the decision Α t Obtained according to the ε-greedy policy based on the environmental state S t at the given time t, Α t = [X(t), H(t)], Α t ∈A, where, Represents the caching decision of the satellite for the cached content of type s1 at time t. When the satellite caches the cached content of type s1 at time t Is 1. When the satellite caches the cached content of type s1 at time t Is 0; Represents the caching decision of the satellite for the cached content of type s2 at time t. When the satellite caches the cached content of type s2 at time t Is 1. When the satellite caches the cached content of type s2 at time t Is 0; Represents the caching decision of the satellite for the cached content of type s3 at time t. When the satellite caches the cached content of type s3 at time t Is 1. When the satellite caches the cached content of type s3 at time t Is 0; H(t) represents the proportion of the satellite requesting cooperation at time t, k represents the identification of neighboring satellites of the satellite, and k ∈ v; Step S3, extract part of the memories from the experience replay space to train and optimize the neural network to obtain the first model.

[0008] The above technical solution: establish a suitable reward function, so that better decisions obtain higher reward values, and endow each LEO satellite node with intelligent decision-making ability through distributed reinforcement learning to judge whether a certain type of cached content should be placed or request cooperation from other nodes, reducing transmission delay and link congestion, effectively improving the ability of LEO satellite payloads to accurately make caching decisions in complex space environments, and achieving optimal allocation of global resources.

[0009] In a preferred embodiment of the present invention, the historical initial cache placement decision of the satellites in the constellation is obtained through a global objective function, and the global objective function is: where t represents the time index, T represents the transmission delay; s represents the cached content; S represents the set of types of cached content, v represents the satellite, and V represents the set of satellites; define as the transmission delay model from satellite v to satellite u; define as the cooperation model, indicating the proportion of the cached content s that requires the cooperation of satellite u under the coverage of satellite v; define as the global cache decision model, indicating whether satellite v caches the cached content s. If the cached content s is placed on satellite v, then If the cached content s is not placed on satellite v, then represents the number of requests for the cached content s under the coverage of satellite v, represents the average transmission delay of the cached content s on the inter-satellite link e i ; represents the inter-satellite link e i whether it is the path between satellite v and u; C s represents the cost of the cached content; ω represents the first weight factor, γ represents the second weight factor, and D s represents the average delay for the cached content s to reach the computing cloud.

[0010] The above technical solution: based on this global objective function, the historical initial cache placement decisions obtained by each satellite in the constellation are optimal.

[0011] In a preferred embodiment of the present invention, the ground control center periodically fuses the received first models based on federated learning to obtain a shared model, which specifically includes: for the K-th round of fusion, a part of the satellites are extracted from the constellation, and the first models of the extracted satellites are uploaded to the ground control center. The ground control center fuses the model parameters of the received first models according to the following formula to obtain the model parameters Q of the shared model obtained in the K-th round of fusion. K : where N represents the total number of satellites extracted in the K-th round of fusion. represents the model parameters of the first model of the i-th satellite among the satellites extracted in the K-th round of fusion.

[0012] The above technical solution: The models are fused through federated learning, and this fusion method can inherit the advantages of each first model and quickly obtain a shared model. In a preferred embodiment of the present invention, when there are conflicts in the decisions output by the private models of the satellites in the constellation, the following is executed: priorities are assigned to the satellites according to the space link conditions, beam coverage areas, and request densities; when the priorities of the satellites with decision conflicts are different, the satellites with high priorities execute the decisions and consume the corresponding resources, and the satellites with low priorities update the environmental information and decisions after the satellites with high priorities execute the decisions.

[0013] The above technical solution: A conflict mitigation mechanism is established to give priority assistance to the satellites with high priorities, effectively alleviating the conflicts in the resource allocation process.

[0014] To achieve the above object of the present invention, according to the second aspect of the present invention, the present invention provides a satellite, which executes: Step 1, obtain the current environmental state of itself, input the current environmental state into the private model of the satellite, and the private model of the satellite outputs a decision on whether to place a cache or request cooperation from other satellite nodes; the environmental state includes the number of various types of requests faced by the satellite, the cache placement status of the satellites adjacent to the satellite, and the remaining processing capacity of the satellite; Step 2, obtain the first model of the satellite through training based on distributed reinforcement learning, the satellite uploads the first model to the ground control center, the ground control center periodically fuses the received first models based on federated learning to obtain a shared model, and the ground control center distributes the shared model to each satellite as the private model of the satellite.

[0015] The above technical solution: The satellite makes cache placement decisions based on the environmental state using the private model. Through distributed reinforcement learning, each low-earth orbit satellite node is given the ability of intelligent decision-making to judge whether a certain type of cache content should be placed or cooperation should be requested from other nodes, reducing the transmission delay and link congestion, effectively improving the ability of the low-earth orbit satellite payload to accurately make cache decisions in a complex space environment, and achieving the optimal allocation of global resources.

[0016] To achieve the above object of the present invention, according to the third aspect of the present invention, the present invention provides a low-earth orbit constellation cache placement system, including a plurality of satellites and a ground control center; the satellites perform: obtaining their own current environmental state, inputting the current environmental state into the private model of the satellite, and the private model of the satellite outputs a decision on whether to place a cache or request cooperation from other satellite nodes; the environmental state includes the number of various types of requests faced by the satellite, the cache placement status of the satellites adjacent to the satellite, and the remaining processing capacity of the satellite; the satellite obtains the first model of the satellite based on distributed reinforcement learning training, and the satellite uploads the first model to the ground control center; the ground control center performs: regularly fusing the received first models based on federated learning to obtain a shared model, and the ground control center distributes the shared model to each satellite as the private model of the satellite.

[0017] The above technical solution: This system proposes a cache placement mechanism based on federated reinforcement learning. First, it models the low-earth orbit constellation network topology and cache placement problem, then trains the autonomous decision-making ability of low-earth orbit satellite nodes through distributed reinforcement learning, and optimizes and trains the low-earth orbit satellite node agents through federated learning strategies. Finally, it solves resource conflicts through a conflict mitigation mechanism, which is beneficial to reducing latency, alleviating link congestion, and improving the dynamic decision-making ability of low-earth orbit satellite node agents. Description of the Drawings

[0018] Figure 1 is a flowchart of a low-earth orbit constellation cache placement method integrating edge computing in a specific embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of the federated learning process in a specific embodiment of the present invention. Specific Embodiments

[0020] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention.

[0021] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0022] In the description of the present invention, unless otherwise specified and defined, it should be noted that the terms "installation", "connection", and "linkage" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the communication inside two components. It can be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific circumstances.

[0023] A method for placing caches in a low-earth orbit constellation integrating edge computing according to the present invention. In a preferred embodiment, the method is as follows: The environmental state of the satellite is obtained in real time and input into the private model of the satellite. The private model of the satellite outputs a decision on satellite cache placement; the decision on cache placement includes the decision result on whether to place a cache on the satellite and the proportion of the satellite requesting cooperation from satellites other than itself in the low-earth orbit constellation; the environmental state includes the number of various types of requests faced by the satellite, the cache placement status of satellites adjacent to the satellite, and the remaining processing capacity of the satellite.

[0024] In this embodiment, preferably, the cached content is an application service. If there is no application service of the requested type on the satellite or the satellite has the application service of this type but the processing capacity is insufficient, this satellite will let other adjacent satellites cooperate in processing the cached content of this requested type and allocate a cooperation processing proportion to each satellite that cooperates in processing. The request types of the cached content preferably but are not limited to including video types and game types. The cache placement status includes that the satellite has placed cached content and has not placed cached content. When the cached content has been placed, the cache placement status is 1, and when the cached content has not been placed, the cache placement status is 0. The remaining processing capacity of adjacent satellites at time t is represented by F(t)=[c1,...,c k , where i∈[0,k], and c i represents the remaining processing capacity of the i-th adjacent satellite.

[0025] In this embodiment, the process of obtaining the private model of satellites in the low-earth orbit constellation includes:

[0026] The satellite obtains a first model of the satellite based on distributed reinforcement learning training, and the satellite uploads the first model to the ground control center; the ground control center regularly fuses the received first models based on federated learning to obtain a shared model, and the ground control center distributes the shared model to each satellite as the private model of the satellite.

[0027] In this embodiment, first, a satellite network topology G(V,E) of the low-earth orbit constellation is constructed, where V represents the set of satellites and E represents the set of inter-satellite links, and each satellite is regarded as a network node.

[0028] In this embodiment, the present invention addresses the problems of long latency and easy link congestion in the low-earth orbit (LEO) constellation accessing the Internet for services, and proposes a cache placement mechanism based on federated reinforcement learning. First, the LEO constellation network topology and cache placement problem are modeled, then the autonomous decision-making ability of LEO satellite nodes is trained through distributed reinforcement learning, and the LEO satellite node agents are optimized and trained through a federated learning strategy. Finally, a conflict mitigation mechanism is used to solve resource conflicts, which is beneficial to reducing latency, alleviating link congestion, and improving the dynamic decision-making ability of LEO satellite node agents.

[0029] In a preferred embodiment, the satellite obtains a first model of the satellite through training based on distributed reinforcement learning, specifically including:

[0030] Step S1, construct the neural network, decision space A, environmental state space S, and reward function R of the satellite t ; Among them, X(t) represents the output value of whether the satellite places the cache decision model at time t; C s represents the cost of the cache content; represents the proportion of the cache content s that requires the cooperation of the neighboring satellite set v of the satellite in the LEO constellation for caching at time t, represents the average transmission delay of the cache content on the inter-satellite link e i ; represents the inter-satellite link e i is a judgment function on whether it is the path between the satellite and the cooperative satellite. When the inter-satellite link e i is the path between the satellite and the cooperative satellite, is equal to 1. When the inter-satellite link e i is not the path between the satellite and the cooperative satellite, is equal to 0. E represents the set of inter-satellite links in the LEO constellation; Penalty represents the penalty term for violating the constraints;

[0031] Step S2, perform memory accumulation based on the ε-greedy policy, and store the memory in the experience replay space. The memory is a vector (S t , A t , R t , S t+1 ), where S t represents the environmental state of the satellite at time t, S t ∈S; S t+1 represents the environmental state of the satellite at time t + 1, S t+1 ∈S; A t represents the decision Α t obtained according to the given environmental state S t at time t according to the ε-greedy policy, Α t= [X(t), H(t)], Α t ∈A, where

[0032] where There are three request types, corresponding to the cached contents s1, s2, and s3 of the three types. Represents the caching decision of the satellite for the cached content of type s1 at time t. When the satellite caches the cached content of type s1 at time t is 1. When the satellite caches the cached content of type s1 at time t is 0; Represents the caching decision of the satellite for the cached content of type s2 at time t. When the satellite caches the cached content of type s2 at time t is 1. When the satellite caches the cached content of type s2 at time t is 0; Represents the caching decision of the satellite for the cached content of type s3 at time t. When the satellite caches the cached content of type s3 at time t is 1. When the satellite caches the cached content of type s3 at time t is 0; H(t) represents the proportion of requests for cooperation by the satellite at time t, k represents the identifier of the neighboring satellite of the satellite, k ∈ v, Represents the proportion of requests for the k-th satellite to cache the cached content of type s1 at time t, Represents the proportion of requests for the k-th satellite to cache the cached content of type s2 at time t, Represents the proportion of requests for the k-th satellite to cache the cached content of type s3 at time t.

[0033] Step S3: Extract part of the memories from the experience replay space to train and optimize the neural network to obtain the first model. The specific process of step S3 is prior art, such as the improved deep reinforcement learning method based on Double DQN disclosed in the Chinese patent with the publication number CN111461321A, which will not be elaborated here. In this embodiment, the satellite continuously repeats steps S2 and S3 until the first model converges or reaches the preset maximum number of training times.

[0034] In a preferred embodiment, the historical initial cache placement decision of the satellites in the constellation is obtained through a global objective function. The global objective function is:

[0035]

[0036] where, t represents the time index, T represents the transmission delay; s represents the cached content; S represents the set of types of cached content, and the type of the cached content s preferably but not limited to includes video or cloud game; v represents a satellite, V represents the set of satellites; define as the transmission delay model from satellite v to satellite u; define as the cooperation model, indicating the proportion of the cached content s that needs the cooperation of satellite u under the coverage of satellite v; define as the global caching decision model, indicating whether satellite v caches the cached content s. If the cached content s is placed on satellite v, then If the cached content s is not placed on satellite v, then represents the number of requests for the cached content s under the coverage of satellite v, represents the average transmission delay of the cached content s on the inter-satellite link e i ; represents whether the inter-satellite link e i is the path between satellite v and u; C s represents the cost of the cached content; ω represents the first weight factor, γ represents the second weight factor, D s represents the average delay for the cached content s to reach the computing cloud.

[0037] In this embodiment, when there is no required service within the area range, the request will be scheduled to the remote cloud, and the proportion of requests allocated to the remote cloud is The average delay for the cached content s to reach the remote cloud is D s , then the transmission delay of the remote cloud is:

[0038] In a preferred embodiment, as Figure 2 shown, the ground control center regularly fuses the received first models based on federated learning to obtain a shared model, specifically including: for the K-th round of fusion, part of the satellites are extracted from the constellation, and the first models of the extracted satellites are uploaded to the ground control center. The ground control center fuses the model parameters of the received first models according to the following formula to obtain the model parameters Q of the shared model obtained in the K-th round of fusion K : where, N represents the total number of satellites extracted in the K-th round of fusion, represents the model parameters of the first model of the i-th satellite among the satellites extracted in the K-th round of fusion. K is a positive integer. In a preferred embodiment, when there are conflicts in the decisions output by the private models of the satellites in the constellation, execute:

[0039] Satellites are assigned priorities according to the space link conditions, beam coverage areas, and request densities. It is judged in turn that the space link condition with high channel quality has a higher priority, then the beam coverage area with a higher priority refers to the urban area, and finally the request density with a higher priority refers to the large number of requests.

[0040] When the priorities of satellites with decision conflicts are different, the satellite with the higher priority executes the decision and consumes the corresponding resources. After the satellite with the lower priority executes the decision, it updates the environmental information and the decision. In this embodiment, since the environmental information that the low-earth orbit satellite agent can obtain is limited. Therefore, the decisions made by a single low-earth orbit satellite agent often cannot take into account the optimal allocation of global resources. When the overall regional load is too large, the decisions of each low-earth orbit satellite agent may conflict, competing for each other's resources, resulting in congestion. Therefore, the above conflict mitigation mechanism is designed. Preferably, the conflict is reflected in the collaborative decision-making. When the remaining processing capacity of the satellite is insufficient to meet the collaborative needs of multiple neighboring satellites, it is judged that there is a resource conflict.

[0041] The present invention proposes a cache placement mechanism based on federated reinforcement learning for the situations of long latency and link congestion when low-earth orbit constellations access the Internet. The whole mechanism mainly includes two parts, namely cache placement based on distributed reinforcement learning and model training based on federated learning. The cache placement based on distributed reinforcement learning mainly considers the total cost such as latency and service cost. On the basis of training an agent model for low-earth orbit satellite nodes, federated learning is adopted to retrain the model through model sharing and model fusion. At the same time, in order to solve the resource conflicts that may be caused by incomplete node information, a conflict mitigation strategy with lower complexity is designed. This method is beneficial to solving the problem that the previous cache placement cannot make intelligent caching and collaborative decisions well for complex environments such as low-earth orbit constellations, and is beneficial to reducing latency, alleviating link congestion, improving the adaptability of agents to the environment, and protecting the privacy and security of node shared data.

[0042] The present invention also discloses a satellite. In a preferred embodiment, the satellite executes:

[0043] Step 1, obtain the current environmental state of itself, input the current environmental state into the private model of the satellite, and the private model of the satellite outputs a decision on whether to place a cache or request cooperation from other satellite nodes; the environmental state includes the number of various types of requests faced by the satellite, the cache placement status of the satellites adjacent to the satellite, and the remaining processing capacity of the satellite;

[0044] Step 2, obtain the first model of the satellite through training based on distributed reinforcement learning. The satellite uploads the first model to the ground control center. The ground control center regularly fuses the received first models based on federated learning to obtain a shared model, and the ground control center distributes the shared model to each satellite as the private model of the satellite.

[0045] The present invention also discloses a cache placement system for a low-earth orbit constellation. In a preferred embodiment, it includes multiple satellites and a ground control center;

[0046] The satellite performs: obtaining its own current environmental state, inputting the current environmental state into the private model of the satellite, and the private model of the satellite outputs a decision on whether to place a cache or request cooperation from other satellite nodes; the environmental state includes the number of various types of requests faced by the satellite, the cache placement status of the satellites adjacent to the satellite, and the remaining processing capacity of the satellite; the satellite obtains the first model of the satellite based on distributed reinforcement learning training, and the satellite uploads the first model to the ground control center;

[0047] The ground control center performs: regularly fusing the received first models based on federated learning to obtain a shared model, and the ground control center distributes the shared model to each satellite as the private model of the satellite.

[0048] A cache placement mechanism based on federated reinforcement learning for a low-earth orbit constellation suitable for integrated edge computing is proposed. The main idea of this mechanism is to endow each low-earth orbit satellite node with intelligent decision-making ability through distributed reinforcement learning to judge whether to place a certain type of cache content or request cooperation from other nodes. Each low-earth orbit satellite node is regarded as an independent agent and makes decisions quickly based on its own environmental information. At the same time, by uploading the shared model obtained through its own training to the ground control center for sharing, and fusing the models through federated learning, and then distributing them to the low-earth orbit satellite nodes for retraining, the purpose of continuously improving the model and continuously learning the decision-making ability of other low-earth orbit satellite nodes is achieved. Therefore, the entire mechanism mainly includes two parts, namely cache placement based on distributed reinforcement learning and model training based on federated learning. Cache placement based on distributed reinforcement learning mainly considers the total cost such as latency and service cost. Based on the shared model trained by the low-earth orbit satellite nodes, federated learning is used to retrain the model through model sharing and model fusion. At the same time, in order to solve the resource conflict that may be caused by incomplete node information, a conflict mitigation strategy with lower complexity is designed. This method is beneficial to solving the problem that the previous cache placement cannot make intelligent cache and cooperation decisions well for complex environments such as low-earth orbit constellations, and is beneficial to reducing latency, alleviating link congestion, improving the adaptability of agents to the environment, and protecting the privacy and security of node shared data.

[0049] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc., mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0050] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for caching placement in a low-earth orbit constellation integrating edge computing, characterized in that Obtain the environmental status of the satellite in real time, input the environmental status of the satellite into the private model of the satellite, and the private model of the satellite outputs the cache placement decision of the satellite; The cache placement decision includes the decision result of whether the satellite places a cache, and the ratio of the satellite requesting cooperation from satellites other than the satellite in the low-earth orbit constellation; The environmental status includes the number of various types of requests faced by the satellite, the cache placement status of satellites adjacent to the satellite, and the remaining processing capacity of the satellite; The process of obtaining the private model of satellites in the low-earth orbit constellation includes: The satellite obtains the first model of the satellite based on distributed reinforcement learning training, and the satellite uploads the first model to the ground control center; The method for the satellite to obtain the first model of the satellite based on distributed reinforcement learning training is specifically: Construct the neural network, decision space A, environmental state space S, and reward function R of the satellite t , perform memory accumulation based on the ε-greedy policy, store the memory in the experience replay space, and extract part of the memory from the experience replay space to train and optimize the neural network to obtain the first model; The ground control center periodically fuses the received first models based on federated learning to obtain a shared model. Specifically, for the K-th round of fusion, part of the satellites are selected from the constellation, and the first models of the selected satellites are uploaded to the ground control center. The ground control center sums and averages the model parameters of the received first models to obtain the model parameters of the shared model obtained in the K-th round of fusion; The ground control center distributes the shared model to each satellite as the private model of the satellite.

2. The low-orbit constellation cache placement method integrating edge computing according to claim 1, wherein The satellite obtains the first model of the satellite based on distributed reinforcement learning training, specifically including: Step S1: Construct the neural network of the satellite, decision space A, environmental state space S, and reward function R t ; where X(t) represents the output value of the satellite's cache placement decision model at time t; C s represents the cost of the cached content; represents the proportion of the cached content s at time t that requires collaborative caching by the neighboring satellite set v of the satellite in the low-earth orbit constellation; Type represents the number of each type of request faced by the satellite; represents the average transmission delay of the cached content over the inter-satellite link e i ; represents the judgment function of whether the inter-satellite link e i is the path between the satellite and the collaborative satellite. When the inter-satellite link e i is the path between the satellite and the collaborative satellite, equals 1. When the inter-satellite link e i is not the path between the satellite and the collaborative satellite, equals 0. E represents the set of inter-satellite links in the low-earth orbit constellation; Penalty represents the penalty term for violating the constraints; Step S2, based on the ε-greedy policy, perform memory accumulation and store the memory in the experience replay space. The memory is a vector (S t , A t , R t , S t+1 ), where S t represents the environmental state of the satellite at time t, S t ∈ S; S t+1 represents the environmental state of the satellite at time t+1, S t+1 ∈ S; A t represents the decision Α t obtained according to the ε-greedy policy based on the environmental state S t at the given time t, Α t = [X(t), H(t)], Α t ∈ A, where represents the caching decision of the satellite at time t for the cached content of type s1. When the satellite caches the cached content of type s1 at time t is 1, and when the satellite caches the cached content of type s1 at time t is 0; represents the caching decision of the satellite at time t for the cached content of type s2. When the satellite caches the cached content of type s2 at time t is 1, and when the satellite caches the cached content of type s2 at time t is 0; represents the caching decision of the satellite at time t for the cached content of type s3. When the satellite caches the cached content of type s3 at time t is 1, and when the satellite caches the cached content of type s3 at time t is 0; H(t) represents the proportion of the satellite requesting cooperation at time t, k represents the identification of the neighboring satellite of the satellite, k ∈ v, represents the proportion of requesting the 1st satellite to cache the cached content of type s1 at time t, represents the proportion of requesting the 1st satellite to cache the cached content of type s2 at time t, represents the proportion of requesting the 1st satellite to cache the cached content of type s3 at time t; Denote the proportion of the cache content of the k-th satellite with cache type s1 requested at time t, Denote the proportion of the cache content of the k-th satellite with cache type s2 requested at time t, Denote the proportion of the cache content of the k-th satellite with cache type s3 requested at time t; Step S3, extract part of the memories from the experience replay space to train and optimize the neural network to obtain the first model.

3. The low-orbit constellation cache placement method integrating edge computing according to claim 1, wherein the constellation The historical initial cache placement decision of the satellite is obtained through the global objective function. The global objective function is: Among them, \(t\) represents the time index, \(T\) represents the transmission delay; \(s\) represents the cached content; \(S\) represents the set of types of cached content, \(v\) represents a satellite, and \(V\) represents the set of satellites; define as the transmission delay model from satellite \(v\) to satellite \(u\); define as the cooperation model, indicating the proportion of the cached content \(s\) that requires the cooperation of satellite \(u\) under the coverage of satellite \(v\); define as the global caching decision model, indicating whether satellite \(v\) caches the content \(s\). If the content \(s\) is placed in satellite \(v\), then If the content \(s\) is not placed in satellite \(v\), then represents the number of requests for the cached content \(s\) under the coverage of satellite \(v\), represents the average transmission delay of the cached content \(s\) on the inter-satellite link \(e\) i ; represents whether the inter-satellite link \(e\) i is the path between satellites \(v\) and \(u\); \(C\) s represents the cost of the cached content; \(\omega\) represents the first weight factor, \(\gamma\) represents the second weight factor, \(D\) s represents the average delay for the cached content \(s\) to reach the computing cloud.

4. The method for caching placement in a low-earth orbit constellation integrating edge computing according to claim 1, wherein The ground control center periodically fuses the received first models based on federated learning to obtain a shared model, specifically including: For the K-th round of fusion, a part of satellites are extracted from the constellation, and the first models of the extracted satellites are uploaded to the ground control center. The ground control center fuses the model parameters of the received first models according to the following formula to obtain the model parameters Q of the shared model obtained in the K-th round of fusion K : Among them, N represents the total number of satellites extracted in the K-th round of fusion, represents the model parameters of the first model of the i-th satellite among the satellites extracted in the K-th round of fusion.

5. The low-orbit constellation cache placement method integrating edge computing according to claim 1, characterized in that When there are conflicts in the decisions output by the private models of satellites in the constellation, execute: Allocate priorities to satellites according to the space link status, beam coverage area, and request density; When the priorities of satellites with decision conflicts are different, the satellite with the higher priority executes the decision and consumes the corresponding resources. After the satellite with the higher priority executes the decision, the satellite with the lower priority updates the environmental information and the decision.

6. A satellite, characterized in that, The satellite is used for: Obtain its own current environmental status, input the current environmental status into the private model of the satellite, and the private model of the satellite outputs the decision of whether the satellite places a cache or requests cooperation from other satellite nodes; the environmental status includes the number of various types of requests faced by the satellite, the cache placement status of satellites adjacent to the satellite, and the remaining processing capacity of the satellite; Obtain the first model of the satellite based on distributed reinforcement learning training, upload the first model to the ground control center, the ground control center periodically fuses the received first models based on federated learning to obtain a shared model, and the ground control center distributes the shared model to each satellite as the private model of the satellite; The method for the satellite to obtain the first model of the satellite based on distributed reinforcement learning training is specifically: Construct the neural network, decision space A, environmental state space S, and reward function R of the satellite t , perform memory accumulation based on the ε-greedy strategy, store the memory in the experience replay space, and extract some memories from the experience replay space to train and optimize the neural network to obtain the first model; The method for the ground control center to regularly fuse the received first models based on federated learning to obtain a shared model is as follows: For the K-th round of fusion, select some satellites from the constellation, upload the first models of the selected satellites to the ground control center, and the ground control center sums and averages the model parameters of the received first models to obtain the model parameters of the shared model obtained in the K-th round of fusion.

7. A low-earth orbit constellation cache placement system, characterized in that, It includes multiple satellites and a ground control center; The satellite is used to: obtain its own current environmental state, input the current environmental state into the private model of the satellite, and the private model of the satellite outputs a decision on whether to place a cache or request cooperation from other satellite nodes; the environmental state includes the number of various types of requests faced by the satellite, the cache placement status of the satellites adjacent to the satellite, and the remaining processing capacity of the satellite; the satellite obtains the first model of the satellite based on distributed reinforcement learning training, and the satellite uploads the first model to the ground control center; The ground control center is used to: regularly fuse the received first models based on federated learning to obtain a shared model, and the ground control center distributes the shared model to each satellite as the private model of the satellite; The method for the satellite to obtain the first model of the satellite based on distributed reinforcement learning training is as follows: Construct the neural network, decision space A, environmental state space S, and reward function R of the satellite t , perform memory accumulation based on the ε-greedy strategy, store the memory in the experience replay space, extract part of the memory from the experience replay space to train and optimize the neural network to obtain the first model; The method for the ground control center to regularly fuse the received first models based on federated learning to obtain a shared model is as follows: For the K-th round of fusion, select some satellites from the constellation, upload the first models of the selected satellites to the ground control center, and the ground control center sums and averages the model parameters of the received first models to obtain the model parameters of the shared model obtained in the K-th round of fusion.

Images