Satellite-ground cooperative service migration method in satellite edge calculation

Abstract

The invention belongs to the field of satellite edge calculation, and discloses a satellite-ground cooperative service migration method in satellite edge calculation. Constructing an information extraction structure for information extraction to obtain key time sequence information; and designing a satellite-ground collaborative service migration multi-agent reinforcement learning algorithm, taking the key time sequence information as an observation value in the satellite-ground collaborative service migration multi-agent reinforcement learning algorithm, and performing reinforcement learning to obtain an optimal service migration strategy. The invention provides an information extraction method combining a time attention mechanism and a bidirectional gating loop unit, so that a time sequence mode in historical data is effectively captured, and the service migration decision-making precision is improved. In combination with collaborative optimization of a ground base station and a low earth orbit satellite network, an Actor-Critic structure and a global Critic mechanism are adopted, cross-network service migration is optimized, migration cost and service interruption time are reduced, and user service experience is improved. The method has remarkable advantages in the aspects of reward value, migration cost and time delay of obtaining service by the user.

Description

Technical Field

[0001] The present invention relates to the field of satellite edge computing technology, and in particular to a satellite-ground collaborative service migration method in satellite edge computing. Background Art

[0002] With the development of satellite edge computing (SEC) technology, service migration plays a vital role in improving system performance, optimizing resource utilization, and ensuring user experience. Service migration usually refers to migrating tasks or services between different computing nodes to achieve load balancing, reduce latency, and improve bandwidth utilization. With the continuous development of communication network technology, especially the combination of ground base stations and satellite networks, the research on service migration has gradually shifted to how to optimize service migration decisions in heterogeneous network environments.

[0003] At present, some studies have attempted to guide service migration decisions by extracting hidden information from historical data. These methods usually rely on surface features extracted from historical data, such as edge server computing load, communication link bandwidth and other factors to make migration decisions. However, these existing methods do not have enough deep data mining and usually fail to fully explore the time series patterns in historical data. For example, existing methods usually rely on static data or ignore the temporal correlation in time series, which leads to low quality of decision input data and suboptimal service migration decisions. At the same time, frequent service migration will increase migration delay and communication delay, seriously affecting system performance and user experience.

[0004] In addition, most of the existing service migration research is focused on ground base station networks or satellite networks. For service migration in a single network environment, the existing optimization algorithms can only achieve relatively ideal results within a single network. However, with the combination of ground base stations and satellite networks, previous studies have failed to fully consider the heterogeneity and complex network topology between the two, and lack effective cross-network collaboration strategies. The rapid changes and topology changes between different networks make service migration more challenging. Especially in the process of cross-network migration, the service interruption time and migration delay are often long, which cannot meet the delay sensitivity requirements for applications with high real-time requirements (such as autonomous driving, virtual reality, etc.). Summary of the invention

[0005] In order to solve the above problems, the present invention designs a satellite-ground collaborative service migration method in satellite edge computing. Based on the technology of temporal attention and BiGRU, the timing patterns in historical data are fully extracted, and then combined with the collaboration of ground base stations and satellite networks, the decision-making accuracy of service migration is improved, the migration delay and communication delay are reduced, and the efficiency and continuity of cross-network service migration are ensured.

[0006] The technical solution of the present invention is as follows: a satellite-ground collaborative service migration method HADDPG in satellite edge computing, constructing an information extraction structure to extract information and obtain key timing information; designing a satellite-ground collaborative service migration multi-agent reinforcement learning algorithm, the key timing information is used as an observation value in the satellite-ground collaborative service migration multi-agent reinforcement learning algorithm, and reinforcement learning is used to obtain the optimal service migration strategy.

[0007] The information extraction structure includes a temporal attention mechanism and a bidirectional gated recurrent unit, which extracts time series patterns in historical data. By introducing attention to key time steps and combining the bidirectional gated recurrent unit to capture bidirectional temporal dependencies, the prediction ability of time series data is improved. The temporal attention mechanism calculates the attention weights of the state and action information of each time step. The bidirectional gated recurrent unit processes data simultaneously from both the past and future directions, capturing long-term and short-term dependencies and trends in historical data.

[0008] The temporal attention mechanism operates as follows:

[0009] a) Critical moment identification and attention weight allocation: During service migration, the temporal attention mechanism calculates the weight of the state information and action information at each time step;

[0010] b) Dynamic weight adjustment and adaptive migration decision: The temporal attention mechanism dynamically adjusts the weight according to the real-time status. When the user is about to leave the current satellite coverage, the attention mechanism will assign a higher weight;

[0011] The temporal attention mechanism selectively focuses on the most critical parts of historical data, captures state correlations in events, and constructs a temporal view that highlights relationships and patterns across time.

[0012] The bidirectional gated recurrent unit includes a forward gated recurrent unit and a backward gated recurrent unit, which performs bidirectional information capture and comprehensive hidden state representation;

[0013] The temporal view is bidirectionally processed through a bidirectional gated recurrent unit, taking into account both past and future dependencies of data, capturing hidden trends and patterns, and effectively integrating temporal information with sequence dependencies;

[0014] During the service migration process, the forward gated loop unit predicts the current state based on the user's past state information and network link changes, and extracts the user's state information at the future time;

[0015] The backward gated recurrent unit starts to process information forward at the last time step of the time series to capture the potential impact of future time steps on the current time step; the backward gated recurrent unit adjusts the service migration decision in advance by predicting future changes in network topology, thereby reducing service interruption time and migration delay;

[0016] By concatenating the hidden states of the forward gated recurrent unit and the backward gated recurrent unit, a comprehensive hidden state representation h is obtained. t , as the observed value.

[0017] The specific processing process of the information extraction structure is as follows:

[0018] Input sequence X = {x1, x2, ..., x T}, contains the state information s for each time step t and action information a t , the input x at each time step t =[s t ; a t ]; In the service migration scenario, the status information s t Including the current user's location, server status information, bandwidth, user access delay, action information a t Represents the action performed by the server at that moment;

[0019] First, a feedforward neural network is used to train the input sequence x at each time step t The attention score is obtained by processing. The specific calculation formula is:

[0020] e t =v T tanh(Wx t +b) (1)

[0021] Among them, v T is the weight vector, W is the weight matrix, and b is the bias vector;

[0022] Convert the attention score to weight: attention score e t Transformed into attention weight α through softmax function t , ensuring that the weights sum to 1, so that each time step contributes a proportion to the total migration decision:

[0023]

[0024] Weighted calculation of temporal attention representation: through attention weight α t Perform a weighted sum on the input sequence,

[0025] Thus we get the temporal attention representation c t :

[0026] c t =α t x t (3)

[0027] The forward gated recurrent unit processes the data from time steps 1 to T and generates the forward hidden state

[0028]

[0029] The backward gated recurrent unit processes the data from time step T to 1 and generates the backward hidden state

[0030]

[0031] The bidirectional gated recurrent unit uses three main gating mechanisms: the update gate z t , reset gate t and candidate hidden states These gating mechanisms are used to control the flow of information and state updates; the specific calculation process is as follows:

[0032] z t =σ(W z c t +U z h t-1 +b z ) (6)

[0033] σ is the Sigmoid activation function, W z and U z is the weight matrix, c t is the current input, h t-1 is the hidden state at the previous moment, b z is a bias term; the update gate controls the balance between the previous state and the current candidate state;

[0034] r t =σ(W r c t +U r h t-1 +b r ) (7)

[0035]

[0036] The forward hidden state and the backward hidden state are concatenated together to form a bidirectional hidden state:

[0037]

[0038] Hidden states serve as observation values ​​for agents in multi-agent reinforcement learning algorithms for service transfer in satellite-ground collaboration.

[0039] The satellite-ground collaborative service migration multi-agent reinforcement learning algorithm includes two local Actor-Critic structures and a global Critic network; the two local Actor-Critic structures are the ground base station Actor-Critic structure and the satellite network Actor-Critic structure;

[0040] The ground base station and the satellite network are heterogeneous in status information; the status information of the ground base station includes the load, bandwidth, computing power, user location, and service demand of the local network; the status information of the satellite network includes the satellite's orbit information, signal quality, coverage, satellite-to-ground link bandwidth, and satellite available computing resources;

[0041] The global Critic network is used to evaluate the overall effect of the local service migration decision and pass the evaluation results to each local Actor-Critic network.

[0042] The satellite-ground collaborative service migration multi-agent reinforcement learning algorithm is specifically:

[0043] In each Actor-Critic structure, the agent updates its strategy through local state and action information to achieve the local optimal decision. By introducing the global critic network, the decisions of all agents are globally evaluated and coordinated. The global critic network comprehensively evaluates the behavior of all agents and provides global feedback to ensure that each agent can take into account the global network performance goals while optimizing locally.

[0044] In the local Actor-Critic structure of ground base stations, each Agent represents a different ground base station and is used to manage service migration decisions in the base station network, taking into account the computing resources, storage resources, bandwidth and user needs of the base station; the Actors are used to generate migration actions of ground base stations, and the Local Critic is used to evaluate the migration decisions of ground base stations; when a user moves from the coverage of one ground base station to the coverage of another ground base station, the possibility of triggering service migration of the base station is determined;

[0045] In the local Actor-Critic structure of the satellite network, it is used to process the migration decision of the satellite network, taking into account the satellite orbit, the transmission capacity between satellites and the limited computing and storage resources; when the satellite connected to the user leaves the coverage area, the service migration between satellites is triggered;

[0046] The global critic network provides global feedback on the migration decisions between ground base stations and satellite networks;

[0047] For ground base stations, the global critic network evaluates the contribution of the service migration decision of the ground base station to the entire network, including service interruption time and delay. For satellite networks, the global critic network evaluates the impact of satellite networks on delay, link quality, and resource consumption during service migration. The global critic network performs global collaborative optimization: the global critic network evaluates the effect of each local decision and feeds back to the respective actors based on the actual network status.

[0048] Based on local reward values ​​and global reward values, the service delay and energy consumption of both the ground base station and the satellite network are evaluated, and their respective local reward values ​​are calculated. The global critic architecture is then used to coordinate the behaviors of the two to minimize the overall user service delay.

[0049] a) Calculation of local reward value of ground base station: When calculating the local reward value of ground base station, two factors, service delay and energy consumption, are considered; the local reward value of ground base station is calculated based on the service delay and energy consumption of the user equipment UE it serves; the lower the service delay and energy consumption, the higher the reward value of the ground base station; the calculation formula is:

[0050]

[0051] Among them, Latency UE Indicates the service delay of the user equipment served by the ground base station, Energy BS represents the energy consumption of the user equipment served by the ground base station; where Latency UE The calculation delay is Communication delay and migration delay Composition, calculated as follows:

[0052]

[0053] Energy BS is the energy consumed by the ground base station calculation Energy consumed in communicating with ground users and the energy consumed by service migration between ground base stations Composition, calculated as follows:

[0054]

[0055] b) Calculation of local reward value of satellite network: When calculating the local reward value of satellite network, two key factors are considered: service delay and energy consumption. The local reward value of satellite network is not only evaluated based on the service delay of the user equipment UE it serves, the shorter the delay, the higher the reward; the energy consumption of the satellite is also considered, the lower the energy consumption, the higher the reward value;

[0056]

[0057] Among them, Latency UE Indicates the service delay of the satellite network server, Energy Sat Energy represents the energy consumption of users of satellite services; Sat is the energy consumed by satellite network computing Energy consumed in communicating with ground users and the energy consumed by service migration between satellite networks Composition, calculated as follows:

[0058]

[0059] c) Reward value calculation of the global critic structure:

[0060] r global =-ζ BS ×Latency BS -τ BS ×Energy BS -η Sat ×Latency Sat -θ Sat ×Energy Sat (16)

[0061] Among them, Latency BS Indicates the service delay of the ground base station, Energy BS Indicates the energy consumption of the ground base station, Latency Sat Indicates the service delay of the satellite network server, Energy Sat Represents the energy consumption of users served by satellite networks.

[0062] Beneficial effects of the present invention: Compared with the service migration algorithm in the existing edge computing, the present invention proposes a satellite-ground collaborative service migration method in satellite edge computing, and combines the time attention mechanism and the bidirectional gated recurrent unit for information extraction. In order to improve the quality of service migration decisions, the present invention improves the existing information extraction method: the time attention mechanism is used to calculate the attention weights of the state and action information of each time step to ensure the prominence of key time series information; combined with BiGRU to capture the long-term and short-term dependencies in historical data, the prediction ability of time series data is comprehensively improved. In this way, the intelligent agent can make more accurate decisions based on the extracted time series information in the local environment. The present invention also introduces heterogeneous design on the state information of ground base stations and satellite networks. The heterogeneous design enables each intelligent agent to make decisions based on its specific environment and needs. At the same time, the global Critic network is used to evaluate the overall effect of local service migration decisions, and the evaluation results are passed to each local Actor-Critic network to ensure the coordination and optimization of local decisions. This design effectively solves the problems of insufficient adaptability and convergence difficulties in traditional algorithms, reduces delays and resource consumption during service migration, and improves the overall performance of the system. After comparative experiments with the baseline algorithm, this algorithm showed significant advantages in reward value, migration cost, algorithm convergence degree and user service latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 Schematic diagram of integrated satellite ground network;

[0064] Figure 2 Schematic diagram of information extraction combining temporal attention and BiGRU;

[0065] Figure 3 It is a schematic diagram of the GRU structure;

[0066] Figure 4 The service migration algorithm architecture diagram for satellite-ground collaboration;

[0067] Figure 5 Comparison chart of final reward values ​​for service migration;

[0068] Figure 6 This is a comparison chart of migration costs during service migration;

[0069] Figure 7 Comparison chart of the convergence degree of algorithms used for service migration;

[0070] Figure 8 A comparison chart of service latency for users during service migration. DETAILED DESCRIPTION

[0071] (1) Information extraction structure: Based on the comprehensive consideration of the composition of state information and the characteristics of the service migration environment, the present invention combines the temporal attention mechanism and BiGRU (bidirectional gated recurrent unit) to improve the accuracy of service migration decisions by extracting timing patterns in historical data. The temporal attention mechanism identifies the most critical moments for the migration task by calculating the attention weights of the state and action information at each time step. BiGRU uses its bidirectional characteristics to capture timing dependencies and fully consider historical and future information, thereby optimizing service migration decisions and reducing migration delays and communication delays. The overall information extraction structure can be divided into two parts, namely identifying key timing and comprehensive information extraction.

[0072] 1) The information extraction structure is the core component responsible for processing time series data in the service migration decision-making process. It has the following main functions: extracting key moments in historical data through the temporal attention mechanism and identifying the most important moments for the current migration task, thereby optimizing migration decisions and improving system performance.

[0073] a) Critical moment identification and attention weight allocation: During the service migration process, the present invention introduces a temporal attention mechanism to calculate the weight of the state information (such as user location information, network status) and action information at each time step. Through this mechanism, the most critical time points in the service migration decision can be accurately identified. For example, when the user is about to leave the current satellite coverage, the temporal attention mechanism will automatically assign a higher weight to this moment based on historical data. This design ensures that service migration can be performed at the best time, thereby avoiding service interruptions caused by delayed migration.

[0074] b) Dynamic weight adjustment and adaptive migration decision: Since the location of satellites and users changes at any time, the importance of different time points will also be different. The temporal attention mechanism can dynamically adjust the weights according to the real-time status to adapt to the changing needs in different environments. This dynamic weight allocation method makes service migration decisions more flexible and can respond to sudden changes in network status in a timely manner. For example, when network conditions are abnormal, the weight adjustment mechanism can quickly optimize the migration strategy to minimize service interruptions.

[0075] 2) Through the bidirectional information capture of forward GRU and backward GRU and their comprehensive hidden state representation, the present invention can more comprehensively understand the timing characteristics of the service migration process, including historical data and predictions of future changes. This method effectively optimizes migration decisions, improves service quality, and reduces migration delays and service interruptions.

[0076] a) Information dependency capture of forward GRU: During the service migration process, the forward GRU (bidirectional gated recurrent unit) can capture the feature information in the past time steps that affects the current decision by learning the user's movement trajectory and network link status. Specifically, the forward GRU uses the user's past state information and network link changes to predict the current state, thereby extracting the user's state information in the future.

[0077] b) Future information dependency capture of backward GRU: Due to the constant changes in satellite and user locations, future changes in network status (such as signal attenuation or changes in transmission rate) will have a significant impact on migration decisions. The backward GRU starts to process information forward at the last time step of the sequence, and can capture the potential impact of future time steps on the current time step. The backward GRU predicts future changes in network topology and adjusts service migration decisions in advance, reducing service interruption time and migration delay, thereby ensuring service stability and continuity.

[0078] c) Comprehensive hidden state representation: The present invention obtains a comprehensive hidden state representation by concatenating the hidden states of the forward GRU and the backward GRU. This representation combines the user's historical state information with the prediction of possible future changes, providing a richer and more comprehensive context information. In service migration decisions, this comprehensive hidden state can help the algorithm evaluate when to migrate services and effectively guide the allocation of computing resources between ground base stations and satellites, thereby improving service quality.

[0079] (2) The temporal attention mechanism identifies key moments and dynamically adjusts the weight of each time step to ensure that migration is performed at the right time. The forward GRU captures the user's historical state information, while the backward GRU predicts future network changes and adjusts the migration strategy in advance. The hidden states of the forward GRU and the backward GRU are combined to provide richer contextual information, optimize migration decisions, reduce service interruptions, and improve migration efficiency.

[0080] Input sequence X = {x1, x2, …, x T} contains the state information s for each time step t and action information a t , that is, the input x at each time step t =[s t ; a t ]. In the service migration scenario, the status information s t Including the current user's location, server status information, bandwidth, user access delay, etc., and action information a tRepresents the action performed by the server at that moment, such as whether to migrate. Through this sequence of historical data, the system can infer future network needs based on past network and user status information, and then make accurate service migration decisions.

[0081] a) Temporal Attention Mechanism: The core purpose of the temporal attention mechanism is to calculate the attention weight according to the importance of each time step. In the service migration scenario, different time points have different impacts on the migration decision. For example, when the user is about to leave the coverage of the current satellite, this moment is more critical than other moments, and a higher weight needs to be assigned to this moment so that service migration can be performed in advance.

[0082] To calculate the attention score at each time step, we first use a feed-forward neural network to train the input sequence x at each time step. t The specific calculation formula is:

[0083] e t =v T tanh(Wx t +b) (1)

[0084] Among them, v T is the weight vector, W is the weight matrix, and b is the bias vector. This process is equivalent to evaluating the importance of each time step to service migration and generating an attention score that represents the contribution of that moment to the entire migration decision process.

[0085] b) Convert the attention score into weights: Next, the attention score e t Need to be converted into attention weight α through the softmax function t , ensuring that the weights sum to 1, so that each time step contributes a proportion to the overall migration decision:

[0086]

[0087] c) Weighted calculation of temporal attention representation: Finally, by using the obtained attention weight α t Perform a weighted summation on the input sequence to obtain the temporal attention representation c t :

[0088] c t =α t x t (3)

[0089] This weighting combines historical information and current state information, and assigns different weights according to the importance of each time step.

[0090] d) Processing through BiGRU: BiGRU can fully understand the temporal dependencies in sequence data, taking into account both past information and possible future changes, thereby improving the model's predictive ability. The forward GRU processes data from time steps 1 to T and generates a forward hidden state

[0091]

[0092] The backward GRU processes the data from time step T to 1 and generates the backward hidden state

[0093]

[0094] The forward and backward hidden states are concatenated together to form a bidirectional hidden state:

[0095]

[0096] BiGRU uses three main gating mechanisms: update gate z t , reset gate t and candidate hidden states These gating mechanisms are used to control the flow of information and state updates. The specific calculation process is as follows:

[0097] z t =σ(W z c t +U z h t-1 +b z ) (7)

[0098] r t =σ(W r c t +U r h t-1 +b r ) (8)

[0099]

[0100]

[0101] Through the above gating mechanism, BiGRU can flexibly selectively memorize and forget information, thereby effectively handling long-term and short-term dependency problems in time series data.

[0102] Design a multi-agent reinforcement learning algorithm for satellite-ground collaborative service migration and formulate a satellite-ground collaborative service migration strategy:

[0103] Existing service migration research mainly focuses on the optimization of a single network environment (such as ground base stations or satellite networks), and lacks an effective collaborative migration solution between ground base stations and low-orbit satellite networks. Due to the heterogeneity and rapid changes of the two networks, cross-network migration usually leads to longer service interruptions and migration delays, seriously affecting applications with high real-time requirements (such as autonomous driving, virtual reality, etc.). The present invention solves this problem through a multi-agent reinforcement learning algorithm based on heterogeneous deep deterministic policy gradient (H-DDPG). The algorithm optimizes cross-network service migration through multiple Actor-Critic structures and a global Critic mechanism, reducing migration delays and service interruption time.

[0104] 1) In response to the lack of collaborative migration strategies between ground base stations and low-orbit satellites in satellite edge computing networks, the satellite-ground collaborative service migration multi-agent reinforcement learning algorithm, as the core optimization component, is responsible for solving the challenges of cross-network collaborative migration. The algorithm has the following key functions: by adopting two Actor-Critic structures, it handles local service migration decisions of ground base stations and satellite networks respectively; at the same time, it introduces a global Critic to evaluate the overall effect of local decisions and pass the feedback results to each local Actor-Critic network. Through this satellite-ground collaborative optimization strategy, the algorithm can effectively reduce service interruption time and migration delay during cross-network migration, ensuring a low-latency user service experience.

[0105] a) Ground Base Station Actor-Critic: Each Agent has an Actor and a Local Critic structure. Each Agent represents a different ground base station. The Actor is used to generate the migration action of the ground base station, and the Local Critic is used to evaluate the migration decision of the ground base station. When a user moves from the coverage of one ground base station to the coverage of another ground base station, it may trigger the service migration of the base station.

[0106] b) Satellite Network Actor-Critic: It handles the migration decisions of the satellite network. When the low-orbit satellite to which the user is connected is about to leave the coverage area, it may trigger the service migration between satellites.

[0107] c) Global Critic Network: Based on the data in the Global Replay Buffer, it comprehensively evaluates the migration decisions between ground base stations and satellite networks and provides global feedback. The global Critic network feeds back the global evaluation results to each local Actor-Critic network, guiding them to optimize the local migration strategy, making local decisions more efficient and coordinated globally.

[0108] 2) In the satellite edge computing network, the design of the collaborative migration strategy between ground base stations and satellites needs to fully consider the heterogeneity of the two. By designing independent Actor-Critic architectures for ground base stations and low-orbit satellites respectively, and combining them with a global Critic architecture, the heterogeneity problem can be effectively solved and the overall performance can be improved.

[0109] a) Different mission requirements: The main task of ground base stations is to provide services for users moving within a short distance, with a focus on resource scheduling and load balancing to ensure low-latency response. Ground base stations usually deal with resource allocation issues for high-density users and quickly adjust resources based on user locations. Satellite networks need to serve a wide area, and face complex environments with high-speed user movement and wide coverage, making their tasks more diverse and challenging. Satellites need to frequently switch communication links while overcoming high latency, bandwidth limitations, and signal attenuation.

[0110] b) Adaptability: A single Actor-Critic architecture cannot meet the different needs of ground base stations and satellites at the same time. Ground base stations focus on short latency and fast response, while satellites need to deal with complex dynamic environments, such as energy management and frequent link switching. If a unified strategy model is adopted, it will not be able to fully consider the specific mission requirements of both, which may lead to performance degradation and affect the effect of satellite-ground collaborative service migration.

[0111] The above problems can be effectively solved by adopting independent Actor-Critic architectures for ground base stations and satellites, combined with global Critic coordination of resources and task allocation.

[0112] 3) In order to optimize the cross-network service migration decision, the present invention proposes a calculation method based on local reward value and global reward value. By evaluating the service delay and energy consumption of ground base stations and low-orbit satellites, the local reward values ​​of each are calculated, and the global Critic architecture is used to coordinate the behaviors of the two to minimize the overall user service delay, thereby improving the service quality.

[0113] a) Calculation of local reward value of ground base station: When calculating the local reward value of ground base station, the two main factors are service delay and energy consumption. The local reward value of ground base station can be calculated based on the service delay and energy consumption of the user equipment (UE) it serves. The lower the service delay and energy consumption, the higher the reward value of the base station. The calculation formula is:

[0114]

[0115] Among them, Latency UE Indicates the service delay of the user equipment served by the base station, Energy BS Indicates the energy consumption of the user equipment served by the base station.UE The calculation delay is Communication delay and migration delay Composition, calculated as follows:

[0116]

[0117] Where EnergyBS is the energy consumed by the ground base station calculation Energy consumed in communicating with ground users and the energy consumed by service migration between ground base stations Composition, calculated as follows:

[0118]

[0119] b) Low Earth Orbit (LEO) satellites are mainly responsible for large-scale regional coverage and communication. When calculating the local reward value of a low-orbit satellite, two key factors are considered: service delay and energy consumption. The local reward value of a satellite is not only evaluated based on the service delay of the user equipment (UE) it serves, the shorter the delay, the higher the reward; it also takes into account the energy consumption of the satellite, the lower the energy consumption, the higher the reward value.

[0120]

[0121] Among them, Latency UE Indicates the service delay of the satellite server, Energy Sat Energy represents the energy consumption of users of satellite services. Sat is the energy consumed by low-orbit satellite computing Energy consumed in communicating with ground users and the energy consumed by service migration between low-orbit satellites Composition, calculated as follows:

[0122]

[0123] c) Reward calculation of the global critic architecture: The reward calculation of the global critic architecture focuses on the performance of local decisions in the entire system, and mainly focuses on how to coordinate the behavior of ground base stations and low-orbit satellites to minimize the user's service delay. The final global reward calculation formula is:

[0124] r global =-ζ BS ×Latency BS -τ BS ×Energy BS -η Sat ×Latency Sat -θSat ×Energy Sat (16)

[0125] Among them, Latency BS Indicates the service delay of the ground base station, Energy BS Indicates the energy consumption of the ground base station, Latency Sat Indicates the service delay of the satellite server, Energy Sat Represents the energy consumption of users of satellite services.

[0126] Algorithm 1 is the pseudo code of the heterogeneous deep reinforcement learning algorithm.

[0127]

[0128]

[0129] The hardware and software environment of this implementation plan is shown in Table 1.

[0130] Table 1 Software and hardware configuration environment

[0131] name Configuration operating system Ubuntu 18.04 CPU i7-13620h Memory 16G

[0132] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings

[0133] Step 1, such as Figure 2 , Figure 3 As shown in the figure, the structure diagram of the system shows the implementation of satellite-ground collaborative service migration. The entire system is divided into two modules from top to bottom: information extraction module and satellite-ground collaborative service migration optimization module. The information extraction module combines the temporal attention mechanism and BiGRU (bidirectional gated recurrent unit) to identify the most critical moments for the migration task by calculating the attention weights of the state and action information of each time step. BiGRU uses its bidirectional characteristics to capture temporal dependencies and fully consider historical and future information, thereby optimizing service migration decisions and reducing migration delays and communication delays. The satellite-ground collaborative service migration optimization module is a multi-agent reinforcement learning algorithm for service migration equipped with satellite-ground collaboration. It mainly provides specific application functions related to service migration and includes two core modules, the local optimization module and the global collaboration module. The local optimization module designs local reward value calculation and local strategy optimization functions for the heterogeneous characteristics of ground base stations and low-orbit satellites respectively; the global collaboration module realizes global scheduling and collaboration of services through multi-agent reinforcement learning.

[0134] Step 2: Build user movement:

[0135] (1) The user's mobility logic is implemented through dynamic interaction with ground base stations and low-orbit satellites. First, the user obtains the neighbor base station list of the ground base station to which it is currently connected, and adds the ID of the current base station to the neighbor list as one of the potential target base stations. At the same time, it starts to check whether the user is within the coverage of the low-orbit satellite, and then adds the available low-orbit satellite ID to the candidate target list. After that, a target node ID is randomly selected from the candidate target list of neighbor base stations and low-orbit satellites, and the corresponding ground base station or low-orbit satellite is obtained based on the ID.

[0136] If the target node is the same as the current node, the user maintains the same position; if the target node is a different ground base station, it indicates that the user starts to move, and the process is: the user connects to the target base station, disconnects from the current base station, and updates the user's base station reference to the target base station. If the target node is a low-orbit satellite, the user will try to connect to the target satellite, and the process includes switching to connect to the low-orbit satellite, disconnecting from the ground base station, and updating the user's connection to the target satellite. In addition, the user will call the position resampling method to update its specific position coordinates in the ground base station or satellite coverage.

[0137] This multi-level mobility logic combines the neighborhood structure of ground base stations and the wide-area coverage characteristics of low-orbit satellites, ensuring that users can dynamically switch between ground and satellite networks and maintain the continuity of user connections and real-time updates of locations.

[0138] Result analysis: 1) Figure 5 As shown in the figure, the convergence speed becomes faster. In the initial stage (0-100 rounds), HADDPG with an information extraction module obviously converges faster and the reward value rises more rapidly. This shows that with the information extraction module, the algorithm can learn the characteristics of the environment more effectively and adopt a better service migration strategy. This reflects the contribution of the information extraction module because it reduces unnecessary exploration and helps the model focus on more useful information. 2) Higher rewards. Throughout the training process, HADDPG with an information extraction module has always maintained a higher reward value, and finally reached a stable state around 500 rounds, with a final reward value of about -200. In contrast, the reward value of HADDPG without an information extraction module is stable at around -300, which is significantly lower than the former. It proves that the information extraction mechanism improves the service migration strategy, and ultimately enables the agent to learn a better strategy and obtain a higher cumulative reward. From the perspective of the smoothness of the curve, HADDPG with an information extraction module shows better stability in the later training stage (300-500 rounds), and the curve fluctuates less. Although HADDPG without the information extraction module gradually stabilizes, the fluctuation range is slightly larger, indicating that its strategy is less stable in the later stage of training.

[0139] Migration costs Figure 6 , the migration cost is higher at the beginning, but the downward trend is more obvious. In the early stage of training (0-50 rounds), the migration cost of both algorithms is high, the initial cost of DDPG is about 10, and that of HADDPG is about 8.5, indicating that the system is not optimized in the initial stage and there is a high resource migration consumption. However, the migration cost of HADDPG decreases faster, and it quickly converges to a lower cost level compared to DDPG, indicating that the algorithm can identify better strategies earlier to reduce resource waste. 2) The overall migration cost is lower. In the middle and late stages of training (50-500 rounds), the migration cost of the HADDPG algorithm is significantly lower than that of DDPG.

[0140] HADDPG: The migration cost is stable at around 2.5-3.0. DDPG: The migration cost is stable at around 3.5-4.0. This gap shows that HADDPG is more efficient in optimizing task scheduling and resource allocation, reducing the overall resource migration consumption. From the curve fluctuation, the migration cost curve of HADDPG is smoother and the fluctuation is smaller. On the contrary, the DDPG curve fluctuates greatly, especially in the mid-term stage (100-300 rounds), there are obvious repeated fluctuations, indicating that it is still unstable in the optimization of migration strategy and has failed to continuously reduce costs.

[0141] Figure 7 The total loss (Total Loss) changing trend of the two algorithms HADDPG and DDPG in 500 training rounds is shown. The vertical axis represents the total loss, and the horizontal axis represents the number of training rounds (Episodes). HADDPG curve (yellow): The total loss is high in the initial stage, but as the training progresses, the loss decreases rapidly and gradually stabilizes. After 100 rounds, the fluctuation is small, showing good convergence and stability. DDPG curve (red): The total loss is also high in the initial stage and decreases quickly, but the overall loss is high and fluctuates greatly, especially in the later stage of training (after 300 rounds), the fluctuation is more obvious, and the convergence effect is not as good as HADDPG.

[0142] Figure 8The figure shows the changes in the user service latency of the DDPG and HADDPG algorithms in 500 training rounds. The vertical axis represents the latency, and the horizontal axis represents the number of training rounds (Episodes). DDPG curve (solid line): The overall latency is high and fluctuates greatly, especially during the training process, there are obvious peaks, and the maximum latency is close to 0.24, indicating that the DDPG algorithm has poor stability in task processing and the latency performance fluctuates greatly. HADDPG curve (dotted line): The latency is significantly lower than DDPG, with less fluctuations, more stable performance, and an average latency of about 0.15. HADDPG always maintains a low latency level in the training rounds, showing better optimization effect and robustness.

[0143] In summary, the satellite-ground collaborative service migration method in satellite edge computing designed by the present invention has been realized, and the effectiveness of information extraction and the superiority of the algorithm structure of multiple actor critics have been verified.

Claims

1. A satellite-ground collaborative service migration method in satellite edge computing, characterized in that: An information extraction structure is constructed to extract key timing information. A satellite-ground collaborative service migration multi-agent reinforcement learning algorithm is designed. The key timing information is used as the observation value in the satellite-ground collaborative service migration multi-agent reinforcement learning algorithm, and reinforcement learning is used to obtain the optimal service migration strategy.

2. The satellite-to-ground collaborative service migration method in satellite edge computing according to claim 1, characterized in that: The information extraction structure includes a temporal attention mechanism and a bidirectional gated recurrent unit to extract temporal patterns in historical data. By introducing attention to key time steps and combining the bidirectional gated recurrent unit to capture bidirectional temporal dependencies, the prediction ability of temporal data is improved. The temporal attention mechanism calculates the attention weights of the state and action information of each time step. The bidirectional gated recurrent unit processes data simultaneously from the past and future directions respectively, and captures the long-term and short-term dependencies and trends in the historical data.

3. The satellite-to-ground collaborative service migration method in satellite edge computing according to claim 2, characterized in that: The temporal attention mechanism operates as follows: a) Critical moment identification and attention weight allocation: During service migration, the temporal attention mechanism calculates the weight of the state information and action information at each time step; b) Dynamic weight adjustment and adaptive migration decision: The temporal attention mechanism dynamically adjusts the weight according to the real-time status. When the user is about to leave the current satellite coverage, the attention mechanism will assign a higher weight; The temporal attention mechanism selectively focuses on the most critical parts of historical data, captures state correlations in events, and constructs a temporal view that highlights relationships and patterns across time.

4. The satellite-to-ground collaborative service migration method in satellite edge computing according to claim 3 is characterized in that: The bidirectional gated recurrent unit includes a forward gated recurrent unit and a backward gated recurrent unit, which performs bidirectional information capture and comprehensive hidden state representation; The temporal view is bidirectionally processed through a bidirectional gated recurrent unit, taking into account both past and future dependencies of data, capturing hidden trends and patterns, and effectively integrating temporal information with sequence dependencies; During the service migration process, the forward gated loop unit predicts the current state based on the user's past state information and network link changes, and extracts the user's state information at the future time; The backward gated recurrent unit starts to process information forward at the last time step of the time series to capture the potential impact of future time steps on the current time step; The backward gated recurrent unit predicts future changes in network topology and adjusts service migration decisions in advance, reducing service interruption time and migration delay; By concatenating the hidden states of the forward gated recurrent unit and the backward gated recurrent unit, a comprehensive hidden state representation h is obtained. t , as the observed value.

5. The satellite-to-ground collaborative service migration method in satellite edge computing according to claim 4, characterized in that: The specific processing process of the information extraction structure is as follows: Input sequence X = {x1, x2, …, x T }, contains the state information s for each time step t and action information a t , the input x at each time step t =[s t ; a t ]; In the service migration scenario, the status information s t Including the current user's location, server status information, bandwidth, user access delay, action information a t Represents the action performed by the server at that moment; First, a feedforward neural network is used to train the input sequence x at each time step t The attention score is obtained by processing. The specific calculation formula is: in, is the weight vector, W is the weight matrix, and b is the bias vector; Convert the attention score to weight: attention score e t Transformed into attention weight α through softmax function t , ensuring that the weights sum to 1, so that each time step contributes a proportion to the total migration decision: Weighted calculation of temporal attention representation: through attention weight α t Perform a weighted summation on the input sequence to obtain the temporal attention representation c t : c t =a t x t (3) The forward gated recurrent unit processes the data from time steps 1 to T and generates the forward hidden state The backward gated recurrent unit processes the data from time step T to 1 and generates the backward hidden state The bidirectional gated recurrent unit uses three main gating mechanisms: the update gate z t , reset gate t and candidate hidden states These gating mechanisms are used to control the flow of information and state updates; the specific calculation process is as follows: z t =σ(W z c t +U z h t-1 +b z )(6) σ is the Sigmoid activation function, W z and U z is the weight matrix, c t is the current input, h t-1 is the hidden state at the previous moment, b z is the bias term; The update gate controls the balance between the previous state and the current candidate state; r t =σ(W r c t +U r h t-1 +b r )(7) The forward hidden state and the backward hidden state are concatenated together to form a bidirectional hidden state: Hidden states as observation values ​​of agents in multi-agent reinforcement learning algorithms for service transfer in satellite-ground collaboration.

6. The satellite-ground collaborative service migration method in satellite edge computing according to claim 1, characterized in that: The satellite-ground collaborative service migration multi-agent reinforcement learning algorithm includes two local Actor-Critic structures and a global Critic network; the two local Actor-Critic structures are the ground base station Actor-Critic structure and the satellite network Actor-Critic structure; the local Actor-Critic structure is used to process the local service migration decision of the ground base station and the local service migration decision of the satellite network respectively; The ground base station and the satellite network are heterogeneous in status information; the status information of the ground base station includes the load, bandwidth, computing power, user location, and service demand of the local network; the status information of the satellite network includes the satellite's orbit information, signal quality, coverage, satellite-to-ground link bandwidth, and satellite available computing resources; The global Critic network is used to evaluate the overall effect of the local service migration decision and transmit the evaluation result to each local Actor-Critic network.

7. The satellite-ground collaborative service migration method in satellite edge computing according to claim 6, characterized in that: The satellite-ground collaborative service migration multi-agent reinforcement learning algorithm is specifically: In each Actor-Critic structure, the agent updates its strategy through local state and action information to achieve the local optimal decision. By introducing the global critic network, the decisions of all agents are globally evaluated and coordinated. The global critic network comprehensively evaluates the behavior of all agents and provides global feedback to ensure that each agent can take into account the global network performance goals while optimizing locally. In the local Actor-Critic structure of ground base stations, each Agent represents a different ground base station and is used to manage service migration decisions in the base station network, taking into account the computing resources, storage resources, bandwidth and user needs of the base station; the Actors are used to generate migration actions of ground base stations, and the Local Critic is used to evaluate the migration decisions of ground base stations; when a user moves from the coverage of one ground base station to the coverage of another ground base station, the possibility of triggering service migration of the base station is determined; In the local Actor-Critic structure of the satellite network, it is used to process the migration decision of the satellite network, taking into account the satellite orbit, the transmission capacity between satellites and the limited computing and storage resources; when the satellite connected to the user leaves the coverage area, the service migration between satellites is triggered; The global critic network provides global feedback on the migration decisions between ground base stations and satellite networks; For ground base stations, the global critic network evaluates the contribution of the service migration decision of the ground base station to the entire network, including service interruption time and delay. For satellite networks, the global critic network evaluates the impact of satellite networks on delay, link quality, and resource consumption during service migration. The global critic network performs global collaborative optimization: the global critic network evaluates the effect of each local decision and feeds back to the respective actors based on the actual network status.

8. The satellite-ground collaborative service migration method in satellite edge computing according to claim 7, characterized in that: Based on local reward values ​​and global reward values, the service delay and energy consumption of both the ground base station and the satellite network are evaluated, and their respective local reward values ​​are calculated. The global critic architecture is then used to coordinate the behaviors of the two to minimize the overall user service delay. a) Calculation of local reward value of ground base station: When calculating the local reward value of ground base station, two factors, service delay and energy consumption, are considered; the local reward value of ground base station is calculated based on the service delay and energy consumption of the user equipment UE it serves; the lower the service delay and energy consumption, the higher the reward value of the ground base station; the calculation formula is: Among them, Latency UE Indicates the service delay of the user equipment served by the ground base station, Energy BS represents the energy consumption of the user equipment served by the ground base station; where Latency UE The calculation delay is Communication delay and migration delay Composition, calculated as follows: Energy BS is the energy consumed by the ground base station calculation Energy consumed in communicating with ground users and the energy consumed by service migration between ground base stations Composition, calculated as follows: b) Calculation of local reward value of satellite network: When calculating the local reward value of satellite network, two key factors are considered: service delay and energy consumption. The local reward value of satellite network is not only evaluated based on the service delay of the user equipment UE it serves, the shorter the delay, the higher the reward; the energy consumption of the satellite is also considered, the lower the energy consumption, the higher the reward value; Among them, Latency UE Indicates the service delay of the satellite network server, Energy Sat Energy represents the energy consumption of users of satellite services; Sat is the energy consumed by satellite network computing Energy consumed in communicating with ground users and the energy consumed by service migration between satellite networks Composition, calculated as follows: c) Reward value calculation of the global critic structure: r global =-ζ BS ×Latency BS -t BS ×Energy BS -or Sat ×Latency Sat -θ Sat ×Energy Sat (16) Among them, Latency BS Indicates the service delay of the ground base station, Energy BS Indicates the energy consumption of the ground base station, Latency Sat Indicates the service delay of the satellite network server, Energy Sat Represents the energy consumption of users served by satellite networks.

Images