A satellite-ground cooperative service migration method in satellite edge computing

Abstract

The present application belongs to the field of satellite edge computing, and discloses a method for satellite-ground cooperative service migration in satellite edge computing. An information extraction structure is constructed to extract information to obtain key timing information; a multi-agent reinforcement learning algorithm for satellite-ground cooperative service migration is designed, the key timing information is used as an observation value in the multi-agent reinforcement learning algorithm for satellite-ground cooperative service migration, and an optimal service migration strategy is obtained through reinforcement learning. The present application proposes an information extraction method combining a time attention mechanism and a bidirectional gated recurrent unit, effectively captures the timing pattern in historical data, and improves the service migration decision accuracy. In combination with the cooperative optimization of ground base stations and low-orbit satellite networks, the Actor-Critic structure and the global Critic mechanism are adopted to optimize the cross-network service migration, reduce the migration cost and service interruption time, and improve the user service experience. The present application has significant advantages in terms of reward value, migration cost and user service delay.

Description

Technical Field

[0001] The present invention relates to the field of satellite edge computing technology, and in particular to a satellite-ground collaborative service migration method in satellite edge computing. Background Art

[0002] With the development of satellite edge computing (SEC) technology, service migration plays a crucial role in improving system performance, optimizing resource utilization, and ensuring user experience. Service migration typically involves migrating tasks or services between different computing nodes to achieve goals such as load balancing, reducing latency, and improving bandwidth utilization. With the continuous advancement of communication network technology, especially the integration of ground base stations and satellite networks, research on service migration has gradually shifted to optimizing service migration decisions in heterogeneous network environments.

[0003] Currently, some research has attempted to guide service migration decisions by extracting hidden information from historical data. These methods typically rely on surface features extracted from historical data, such as edge server computing load, communication link bandwidth, and other factors, to make migration decisions. However, these existing methods lack deep data mining and often fail to fully explore the time series patterns in historical data. For example, existing methods often rely on static data or ignore temporal correlations in time series, which leads to low-quality decision input data and suboptimal service migration decisions. At the same time, frequent service migrations increase migration latency and communication delays, seriously affecting system performance and user experience.

[0004] In addition, most existing service migration research focuses on ground base station networks or satellite networks. For service migration in a single network environment, existing optimization algorithms can only achieve relatively ideal results within a single network. However, with the combination of ground base stations and satellite networks, previous studies have failed to fully consider the heterogeneity and complex network topology between the two, and lack effective cross-network collaboration strategies. The rapid changes and topology changes between different networks make service migration more challenging. In particular, during cross-network migration, service interruption time and migration delay are often long, which cannot meet the delay sensitivity requirements for applications with high real-time requirements (such as autonomous driving, virtual reality, etc.). Summary of the Invention

[0005] In order to solve the above problems, the present invention designs a satellite-ground collaborative service migration method in satellite edge computing. Based on the technology of temporal attention and BiGRU, it fully extracts the timing patterns in historical data, and combines the collaboration between ground base stations and satellite networks to improve the decision-making accuracy of service migration, reduce migration delay and communication delay, and ensure the efficiency and continuity of cross-network service migration.

[0006] The technical solution of the present invention is as follows: a satellite-ground collaborative service migration method HADDPG in satellite edge computing is constructed to extract information through an information extraction structure to obtain key timing information; a satellite-ground collaborative service migration multi-agent reinforcement learning algorithm is designed, wherein the key timing information is used as an observation value in the satellite-ground collaborative service migration multi-agent reinforcement learning algorithm, and reinforcement learning is used to obtain the optimal service migration strategy.

[0007] The information extraction structure includes a temporal attention mechanism and a bidirectional gated recurrent unit, which extracts time series patterns in historical data. By introducing attention to key time steps and combining the bidirectional gated recurrent unit to capture bidirectional temporal dependencies, the prediction ability of time series data is improved. The temporal attention mechanism calculates the attention weights of the state and action information of each time step. The bidirectional gated recurrent unit processes data simultaneously from both the past and future directions, capturing long-term and short-term dependencies and trends in historical data.

[0008] The temporal attention mechanism operates as follows:

[0009] a) Critical moment identification and attention weight allocation: During service migration, the temporal attention mechanism calculates the weights of the state information and action information at each time step;

[0010] b) Dynamic weight adjustment and adaptive migration decision-making: The temporal attention mechanism dynamically adjusts weights based on real-time status. When a user is about to leave the current satellite coverage area, the attention mechanism assigns a higher weight;

[0011] The temporal attention mechanism selectively focuses on the most critical parts of historical data, captures state correlations in events, and constructs a temporal view that highlights relationships and patterns across time.

[0012] The bidirectional gated recurrent unit includes a forward gated recurrent unit and a backward gated recurrent unit, which performs bidirectional information capture and comprehensive hidden state representation;

[0013] The temporal view performs bidirectional processing through a bidirectional gated recurrent unit, simultaneously considering the past and future dependencies of the data, capturing hidden trends and patterns, and effectively integrating temporal information with sequence dependencies.

[0014] During the service migration process, the forward gated recurrent unit predicts the current state based on the user's past state information and network link changes, and extracts the user's state information at the future time;

[0015] The backward gated recurrent unit starts processing information forward from the last time step of the time series to capture the potential impact of future time steps on the current time step. The backward gated recurrent unit adjusts service migration decisions in advance by predicting future changes in network topology, thereby reducing service interruption time and migration delay.

[0016] By concatenating the hidden states of the forward gated recurrent unit and the backward gated recurrent unit, a comprehensive hidden state representation h is obtained. t , as the observed value.

[0017] The specific processing process of the information extraction structure is as follows:

[0018] Input sequence X = {x1, x2, ..., x T}, contains the state information s of each time step t and action information a t , the input x at each time step t =[s t ;a t ]; In the service migration scenario, the status information s t Including the current user's location, server status information, bandwidth, user access delay, action information a t Represents the action performed by the server at that moment;

[0019] First, a feedforward neural network is used to train the input sequence x at each time step. t The attention score is obtained by processing. The specific calculation formula is:

[0020] e t =v T tanh(Wx t +b) (1)

[0021] Among them, v T is the weight vector, W is the weight matrix, and b is the bias vector;

[0022] Convert the attention score into a weight: attention score e t Converted into attention weight α through softmax function t , ensuring that the weights sum to 1, so that each time step contributes a proportion to the total migration decision:

[0023]

[0024] Weighted calculation of temporal attention representation: by attention weight α t Perform weighted summation on the input sequence,

[0025] Thus we get the temporal attention representation c t :

[0026] c t =α t x t (3)

[0027] The forward gated recurrent unit processes data from time steps 1 to T and generates a forward hidden state

[0028]

[0029] The backward gated recurrent unit processes the data from time step T to 1 and generates the backward hidden state

[0030]

[0031] The bidirectional gated recurrent unit uses three main gating mechanisms: the update gate z t , reset gate r t and candidate hidden states These gating mechanisms are used to control the flow of information and state updates; the specific calculation process is as follows:

[0032] z t =σ(W z c t +U z h t-1 +b z ) (6)

[0033] σ is the Sigmoid activation function, W z and U z is the weight matrix, c t is the current input, h t-1 is the hidden state at the previous moment, b z Is the bias term; the update gate controls the balance between the previous state and the current candidate state;

[0034] r t =σ(W r c t +U r h t-1 +b r ) (7)

[0035]

[0036] The forward hidden state and the backward hidden state are concatenated together to form a bidirectional hidden state:

[0037]

[0038] Hidden states serve as observation values ​​for agents in multi-agent reinforcement learning algorithms for service transfer in space-ground collaboration.

[0039] The satellite-ground collaborative service migration multi-agent reinforcement learning algorithm includes two local actor-critic structures and a global critic network; the two local actor-critic structures are the ground base station actor-critic structure and the satellite network actor-critic structure;

[0040] The ground base station and satellite network are heterogeneous in terms of status information; the status information of the ground base station includes the load, bandwidth, computing power, user location, and service requirements of the local network; the status information of the satellite network includes the satellite's orbit information, signal quality, coverage, satellite-to-ground link bandwidth, and satellite available computing resources;

[0041] The global critic network is used to evaluate the overall effect of local service migration decisions and pass the evaluation results to each local actor-critic network.

[0042] The multi-agent reinforcement learning algorithm for service migration in satellite-ground collaboration is specifically as follows:

[0043] In each actor-critic structure, agents update their strategies based on local state and action information to achieve local optimal decisions. By introducing a global critic network, all agents' decisions are globally evaluated and coordinated. This global critic network comprehensively evaluates the behavior of all agents and provides global feedback, ensuring that each agent can achieve local optimization while also taking into account global network performance goals.

[0044] In the local Actor-Critic architecture for ground base stations, each agent represents a different ground base station and is used to manage service migration decisions in the base station network, taking into account the base station's computing resources, storage resources, bandwidth, and user needs. Actors are used to generate migration actions for ground base stations, and Local Critic is used to evaluate ground base station migration decisions. When a user moves from the coverage area of ​​one ground base station to the coverage area of ​​another, the possibility of triggering service migration of the base station is determined.

[0045] In the local Actor-Critic architecture of satellite networks, it is used to process migration decisions in satellite networks, taking into account satellite orbits, inter-satellite transmission capabilities, and limited computing and storage resources. When the satellite to which a user is connected leaves the coverage area, service migration between satellites is triggered.

[0046] The global critic network provides global feedback on migration decisions between ground base stations and satellite networks;

[0047] For ground base stations, the global critic network evaluates the contribution of the ground base station's service migration decision to the entire network, including service interruption time and latency. For satellite networks, the global critic network evaluates the impact of the satellite network on latency, link quality, and resource consumption during service migration. The global critic network performs global collaborative optimization: the global critic network evaluates the effect of each local decision and provides feedback to each actor based on the actual network status.

[0048] Based on local and global reward values, the service delay and energy consumption of both the ground base station and the satellite network are evaluated, and their respective local reward values ​​are calculated. A global critic architecture is then used to coordinate the behaviors of the two to minimize the overall user service delay.

[0049] a) Calculation of the local reward value of a ground base station: When calculating the local reward value of a ground base station, two factors, service delay and energy consumption, are considered. The local reward value of a ground base station is calculated based on the service delay and energy consumption of the user equipment (UE) it serves. The lower the service delay and energy consumption, the higher the reward value of the ground base station. The calculation formula is:

[0050]

[0051] Among them, Latency UE Indicates the service delay of the user equipment served by the ground base station, Energy BS Indicates the energy consumption of user equipment served by the ground base station; Latency UE The calculation delay is Communication delay and migration delay Composition, calculated as follows:

[0052]

[0053] Energy BS is the energy consumed by the ground base station calculation Energy consumed in communicating with ground users and the energy consumed by service migration between ground base stations Composition, calculated as follows:

[0054]

[0055] b) Calculation of the local reward value of the satellite network: When calculating the local reward value of the satellite network, two key factors are considered: service latency and energy consumption. The local reward value of the satellite network is evaluated not only based on the service latency of the user equipment (UE) it serves, with shorter latency resulting in higher rewards, but also considering the satellite's energy consumption, with lower energy consumption resulting in higher rewards.

[0056]

[0057] Among them, Latency UE Indicates the service delay of the satellite network server, Energy Sat Energy Sat is the energy consumed by satellite network computing Energy consumed in communicating with ground users and the energy consumed by service migration between satellite networks Composition, calculated as follows:

[0058]

[0059] c) Reward calculation of the global critic structure:

[0060] r global =-ζ BS ×Latency BS -τ BS ×Energy BS -η Sat ×Latency Sat -θ Sat ×Energy Sat (16)

[0061] Among them, Latency BS Indicates the service delay of the ground base station, Energy BS Indicates the energy consumption of the ground base station, Latency Sat Indicates the service delay of the satellite network server, Energy Sat Represents the energy consumption of users served by satellite network.

[0062] The beneficial effects of this invention: Compared to existing service migration algorithms in edge computing, this invention proposes a satellite-ground collaborative service migration method for satellite edge computing, combining a temporal attention mechanism and a bidirectional gated recurrent unit for information extraction. To improve the quality of service migration decisions, this invention improves upon existing information extraction methods: a temporal attention mechanism is employed to calculate the attention weights for state and action information at each time step, ensuring the prominence of critical time series information. Furthermore, a BiGRU is combined with this mechanism to capture long-term and short-term dependencies in historical data, comprehensively enhancing the predictive capabilities of time series data. This allows intelligent agents to make more accurate decisions in their local environments based on the extracted time series information. This invention also introduces a heterogeneous design for the state information of ground base stations and satellite networks, enabling each intelligent agent to make decisions based on its specific environment and needs. Furthermore, a global critic network is used to evaluate the overall effectiveness of local service migration decisions and transmits the evaluation results to each local actor-critic network, ensuring coordination and optimization of local decisions. This design effectively addresses the lack of adaptability and convergence difficulties inherent in traditional algorithms, reduces latency and resource consumption during service migration, and improves overall system performance. After comparative experiments with the baseline algorithm, this algorithm showed significant advantages in reward value, migration cost, algorithm convergence degree and user service acquisition delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 Schematic diagram of the integrated satellite ground network;

[0064] Figure 2 Schematic diagram of information extraction combining temporal attention and BiGRU;

[0065] Figure 3 Schematic diagram of GRU structure;

[0066] Figure 4 The service migration algorithm architecture diagram for satellite-ground collaboration;

[0067] Figure 5 Comparison chart of final reward values ​​for service migration;

[0068] Figure 6 A comparison chart of migration costs during service migration;

[0069] Figure 7 Comparison chart of the convergence degree of algorithms used for service migration;

[0070] Figure 8 A comparison chart of service latency for users during service migration. DETAILED DESCRIPTION

[0071] (1) Information extraction structure: Based on the comprehensive consideration of the composition of state information and the characteristics of the service migration environment, the present invention combines the temporal attention mechanism and BiGRU (bidirectional gated recurrent unit) to improve the accuracy of service migration decisions by extracting timing patterns in historical data. The temporal attention mechanism identifies the most critical moments for the migration task by calculating the attention weights of the state and action information at each time step. BiGRU uses its bidirectional characteristics to capture temporal dependencies and fully considers historical and future information, thereby optimizing service migration decisions and reducing migration delays and communication delays. The overall information extraction structure can be divided into two parts, namely, identifying key timing and comprehensive information extraction.

[0072] 1) The information extraction structure is the core component responsible for processing time series data in the service migration decision-making process. It has the following main functions: extracting key moments in historical data through the temporal attention mechanism and identifying the moments that are most important to the current migration task, thereby optimizing migration decisions and improving system performance.

[0073] a) Critical moment identification and attention weight allocation: During the service migration process, the present invention introduces a temporal attention mechanism to calculate weights for state information (such as user location information and network status) and action information at each time step. This mechanism accurately identifies the most critical time points in service migration decisions. For example, when a user is about to leave the current satellite coverage area, the temporal attention mechanism automatically assigns a higher weight to this moment based on historical data. This design ensures that service migration can be executed at the optimal time, thereby avoiding service interruptions caused by delayed migration.

[0074] b) Dynamic Weight Adjustment and Adaptive Migration Decisions: Since the locations of satellites and users fluctuate constantly, the importance of different points in time can also vary. The temporal attention mechanism dynamically adjusts weights based on real-time conditions, adapting to changing needs in different environments. This dynamic weight allocation makes service migration decisions more flexible and enables timely response to sudden network changes. For example, when network conditions deviate, the weight adjustment mechanism can quickly optimize migration strategies to minimize service interruptions.

[0075] 2) By capturing bidirectional information through forward and backward GRUs and integrating their hidden state representation, this method enables a more comprehensive understanding of the temporal characteristics of service migration, including historical data and predictions of future changes. This approach effectively optimizes migration decisions, improves service quality, and reduces migration latency and service disruption.

[0076] a) Forward GRU Information Dependency Capture: During service migration, the forward GRU (bidirectional gated recurrent unit) learns about user movement trajectories and network link status, capturing feature information from past time steps that influence current decisions. Specifically, the forward GRU leverages past user status information and network link changes to predict the current state, thereby extracting information about the user's future state.

[0077] b) Backward GRU Captures Future Information Dependencies: Due to the constant shifting of satellite and user locations, future changes in network status (such as signal attenuation or transmission rate variations) can significantly impact migration decisions. The backward GRU begins forward processing at the last time step of the sequence, capturing the potential impact of future time steps on the current time step. By predicting future network topology changes, the backward GRU proactively adjusts service migration decisions, minimizing service interruption time and migration delays, thereby ensuring service stability and continuity.

[0078] c) Comprehensive Hidden State Representation: This invention concatenates the hidden states of the forward and backward GRUs to create a comprehensive hidden state representation. This representation combines historical user state information with predictions of possible future changes, providing a richer and more comprehensive context. In service migration decisions, this comprehensive hidden state helps the algorithm assess when to migrate services and effectively guides the allocation of computing resources between ground base stations and satellites, thereby improving service quality.

[0079] (2) The temporal attention mechanism identifies key moments and dynamically adjusts the weights of each time step to ensure that migration occurs at the appropriate time. The forward GRU captures historical user state information, while the backward GRU predicts future network changes and adjusts the migration strategy in advance. Combining the hidden states of the forward and backward GRUs provides richer contextual information, optimizes migration decisions, reduces service interruptions, and improves migration efficiency.

[0080] Input sequence X={x1,x2,…,x T}Contains the state information s for each time step t and action information a t , that is, the input x at each time step t =[s t ;a t ]. In the service migration scenario, the status information s t Including the current user's location, server status information, bandwidth, user access delay, etc., and action information a tThis represents the action the server performed at that moment, such as whether to migrate. By using this sequence of historical data, the system can infer future network needs based on past network and user status information, and make accurate service migration decisions.

[0081] a) Temporal Attention Mechanism: The core purpose of the temporal attention mechanism is to calculate attention weights based on the importance of each time step. In a service migration scenario, different time points have different impacts on the migration decision. For example, when a user is about to leave the current satellite coverage area, this moment is more critical than other moments, and a higher weight should be assigned to this moment to facilitate early service migration.

[0082] To calculate the attention score for each time step, we first use a feedforward neural network to train the input sequence x at each time step. t The specific calculation formula is:

[0083] e t =v T tanh(Wx t +b) (1)

[0084] Among them, v T is the weight vector, W is the weight matrix, and b is the bias vector. This process is equivalent to evaluating the importance of each time step to service migration and generating an attention score that represents the contribution of that moment to the entire migration decision process.

[0085] b) Converting attention scores into weights: Next, the attention score e t Need to be converted into attention weight α through the softmax function t , ensuring that the sum of the weights is 1, so that the contribution of each time step has a proportion in the overall migration decision:

[0086]

[0087] c) Weighted calculation of temporal attention representation: Finally, by using the obtained attention weight α t Perform weighted summation on the input sequence to obtain the temporal attention representation c t :

[0088] c t =α t x t (3)

[0089] This weighting combines historical information and current state information, and assigns different weights according to the importance of each time step.

[0090] d) Processing through BiGRU: BiGRU can fully understand the temporal dependencies in sequence data, taking into account both past information and possible future changes, thereby improving the model's predictive power. The forward GRU processes data from time steps 1 to T and generates a forward hidden state.

[0091]

[0092] The backward GRU processes the data from time step T to 1 and generates the backward hidden state

[0093]

[0094] The forward and backward hidden states are concatenated together to form a bidirectional hidden state:

[0095]

[0096] BiGRU uses three main gating mechanisms: update gate z t , reset gate r t and candidate hidden states These gating mechanisms are used to control information flow and state updates. The specific calculation process is as follows:

[0097] z t =σ(W z c t +U z h t-1 +b z ) (7)

[0098] r t =σ(W r c t +U r h t-1 +b r ) (8)

[0099]

[0100]

[0101] Through the above-mentioned gating mechanism, BiGRU can flexibly selectively memorize and forget information, thereby effectively handling long-term and short-term dependency problems in time series data.

[0102] Design a multi-agent reinforcement learning algorithm for space-ground collaborative service migration and formulate a space-ground collaborative service migration strategy:

[0103] Existing service migration research mainly focuses on the optimization of a single network environment (such as ground base stations or satellite networks), and lacks an effective collaborative migration solution between ground base stations and low-orbit satellite networks. Due to the heterogeneity and rapid changes of the two networks, cross-network migration usually leads to longer service interruptions and migration delays, which seriously affects applications with high real-time requirements (such as autonomous driving, virtual reality, etc.). The present invention solves this problem through a multi-agent reinforcement learning algorithm based on heterogeneous deep deterministic policy gradient (H-DDPG). The algorithm optimizes cross-network service migration through multiple Actor-Critic structures and a global Critic mechanism, reducing migration delays and service interruption time.

[0104] 1) To address the lack of a coordinated migration strategy between ground base stations and low-orbit satellites in satellite edge computing networks, a multi-agent reinforcement learning algorithm for coordinated service migration between ground and satellites is used as a core optimization component to address the challenges of cross-network coordinated migration. This algorithm has several key features: It employs two actor-critic structures to handle local service migration decisions for ground base stations and satellite networks, respectively; it also introduces a global critic to evaluate the overall effectiveness of local decisions and transmits feedback to each local actor-critic network. Through this coordinated optimization strategy between ground and satellite, the algorithm effectively reduces service interruption time and migration delays during cross-network migration, ensuring a low-latency user service experience.

[0105] a) Ground Base Station Actor-Critic: Each agent consists of an actor and a local critic. Each agent represents a different ground base station. The actor is used to generate migration actions for the ground base station, and the local critic is used to evaluate the ground base station's migration decisions. When a user moves from one ground base station's coverage area to another, it may trigger a base station's service migration.

[0106] b) Satellite Network Actor-Critic: This handles migration decisions in the satellite network. When the low-orbit satellite to which the user is connected is about to leave the coverage area, it may trigger service migration between satellites.

[0107] c) Global Critic Network: Based on data from the Global Replay Buffer, it comprehensively evaluates migration decisions between ground base stations and the satellite network, providing global feedback. The global critic network feeds back global evaluation results to each local actor-critic network, guiding them to optimize their local migration strategies, making local decisions more efficient and coordinated globally.

[0108] 2) In satellite edge computing networks, the design of collaborative migration strategies between ground base stations and satellites must fully consider the heterogeneity of the two. By designing independent actor-critic architectures for ground base stations and low-orbit satellites, combined with a global critic architecture, this heterogeneity issue can be effectively addressed and overall performance improved.

[0109] a) Different mission requirements: Ground base stations primarily serve users moving over short distances, focusing on resource scheduling and load balancing to ensure low-latency responses. Ground base stations typically handle resource allocation for a high density of users and rapidly adjust resources based on user location. Satellite networks, on the other hand, must serve a wide area. Faced with complex environments characterized by high-speed user movement and extensive coverage, their missions are more diverse and challenging. Satellites must frequently switch communication links while overcoming high latency, bandwidth limitations, and signal attenuation.

[0110] b) Adaptability: A single actor-critic architecture cannot simultaneously meet the diverse needs of both ground base stations and satellites. Ground base stations prioritize low latency and rapid response, while satellites must handle complex dynamic environments, such as energy management and frequent link handoffs. A unified strategy model would fail to fully account for the specific mission requirements of both, potentially leading to performance degradation and compromising the effectiveness of satellite-ground collaborative service migration.

[0111] The above problems can be effectively solved by adopting independent Actor-Critic architectures for ground base stations and satellites, and combining them with global Critic coordination of resources and task allocation.

[0112] 3) To optimize cross-network service migration decisions, this paper proposes a calculation method based on local and global rewards. By evaluating factors such as service latency and energy consumption of ground base stations and low-orbit satellites, local rewards are calculated for each. A global critic architecture is then used to coordinate their behavior, minimizing overall user service latency and thus improving service quality.

[0113] a) Calculation of Local Reward Values ​​for Ground Base Stations: When calculating the local reward value for a ground base station, two factors are primarily considered: service latency and energy consumption. The local reward value of a ground base station can be calculated based on the service latency and energy consumption of the user equipment (UE) it serves. The lower the service latency and energy consumption, the higher the base station's reward value. The calculation formula is:

[0114]

[0115] Among them, Latency UE Indicates the service delay of the user equipment served by the base station, Energy BS Indicates the energy consumption of the user equipment served by the base station.UE The calculation delay is Communication delay and migration delay Composition, calculated as follows:

[0116]

[0117] Where EnergyBS is the energy consumed by the ground base station calculation Energy consumed in communicating with ground users and the energy consumed by service migration between ground base stations Composition, calculated as follows:

[0118]

[0119] b) Low Earth Orbit (LEO) satellites primarily provide large-scale regional coverage and communications. When calculating the local reward for LEO satellites, two key factors are considered: service latency and energy consumption. A satellite's local reward is evaluated not only based on the service latency to the user equipment (UE) it serves—the shorter the latency, the higher the reward—but also based on the satellite's energy consumption: the lower the energy consumption, the higher the reward.

[0120]

[0121] Among them, Latency UE Indicates the service delay of the satellite server, Energy Sat Energy represents the energy consumption of users of satellite services. Sat is the energy consumed by low-orbit satellite computing Energy consumed in communicating with ground users and the energy consumed by service migration between low-orbit satellites Composition, calculated as follows:

[0122]

[0123] c) Reward calculation of the global critic architecture: The reward calculation of the global critic architecture focuses on the performance of local decisions in the entire system, mainly focusing on how to coordinate the behavior of ground base stations and low-orbit satellites to minimize user service delays. The final global reward calculation formula is:

[0124] r global =-ζ BS ×Latency BS -τ BS ×Energy BS -η Sat ×Latency Sat -θSat ×Energy Sat (16)

[0125] Among them, Latency BS Indicates the service delay of the ground base station, Energy BS Indicates the energy consumption of the ground base station, Latency Sat Indicates the service delay of the satellite server, Energy Sat Represents the energy consumption of users of satellite services.

[0126] Algorithm 1 is the pseudo code of the heterogeneous deep reinforcement learning algorithm.

[0127]

[0128]

[0129] The hardware and software environment of this implementation plan is shown in Table 1.

[0130] Table 1 Software and hardware configuration environment

[0131] name Configuration operating system Ubuntu 18.04 CPU i7-13620h Memory 16G

[0132] The specific implementation of the present invention will be described in detail below with reference to the accompanying drawings

[0133] Step 1, such as Figure 2 、 Figure 3 The system architecture diagram shown in Figure 2 illustrates the implementation of satellite-ground collaborative service migration. The entire system is divided into two modules from top to bottom: an information extraction module and a satellite-ground collaborative service migration optimization module. The information extraction module combines a temporal attention mechanism with a BiGRU (bidirectional gated recurrent unit) to calculate the attention weights of state and action information at each time step, identifying the most critical moments for the migration task. The BiGRU leverages its bidirectional nature to capture temporal dependencies and fully consider historical and future information, thereby optimizing service migration decisions and reducing migration latency and communication delays. The satellite-ground collaborative service migration optimization module, which implements a satellite-ground collaborative service migration multi-agent reinforcement learning algorithm, primarily provides specific application functions related to service migration. It comprises two core modules: a local optimization module and a global collaboration module. The local optimization module, targeting the heterogeneous nature of ground base stations and low-orbit satellites, features local reward calculation and local policy optimization, respectively. The global collaboration module implements multi-agent reinforcement learning to achieve global service scheduling and collaboration.

[0134] Step 2: Build user mobility:

[0135] (1) The user's mobility logic is implemented through dynamic interaction with ground base stations and low-orbit satellites. First, the user obtains the neighbor base station list of the currently connected ground base station and adds the ID of the current base station to the neighbor list as one of the potential target base stations. At the same time, it starts to check whether the user is within the coverage of the low-orbit satellite and then adds the available low-orbit satellite IDs to the candidate target list. After that, the ID of a target node is randomly selected from the candidate target lists of neighbor base stations and low-orbit satellites, and the corresponding ground base station or low-orbit satellite is obtained based on the ID.

[0136] If the target node is the same as the current node, the user maintains their position. If the target node is a different ground base station, the user begins to move, and the process involves connecting to the target base station, disconnecting from the current base station, and updating the user's base station reference to the target base station. If the target node is a low-Earth orbit satellite, the user attempts to connect to the target satellite, which involves switching to the low-Earth orbit satellite, disconnecting from the ground base station, and updating the user's connection to the target satellite. Furthermore, the user invokes a position resampling method to update their specific location coordinates within the ground base station or satellite coverage area.

[0137] This multi-level mobility logic combines the neighborhood structure of ground base stations and the wide-area coverage characteristics of low-orbit satellites, ensuring that users can dynamically switch between ground and satellite networks and maintain the continuity of user connections and real-time updates of their locations.

[0138] Result analysis: 1) Figure 5 As shown, convergence speed is faster. In the initial stage (epochs 0–100), HADDPG with the information extraction module converges significantly faster, and the reward value rises more rapidly. This demonstrates that the information extraction module enables the algorithm to more effectively learn the characteristics of the environment and adopt a more optimal service migration strategy. This demonstrates the contribution of the information extraction module, as it reduces unnecessary exploration and helps the model focus on more useful information. 2) The reward is higher. Throughout training, HADDPG with the information extraction module maintains a higher reward value, eventually reaching a stable state around 500 episodes, with a final reward value of approximately -200. In contrast, the reward value of HADDPG without the information extraction module remains stable at around -300, significantly lower than the former. This demonstrates that the information extraction mechanism improves the service migration strategy, ultimately enabling the agent to learn a more optimal strategy and achieve higher cumulative rewards. Judging from the smoothness of the curve, HADDPG with the information extraction module exhibits better stability and less fluctuation in the later stages of training (epochs 300–500). Although HADDPG without the information extraction module gradually stabilizes, the fluctuation range is slightly larger, indicating that its strategy is less stable in the later stages of training.

[0139] Migration costs Figure 6 , the migration cost is initially high, but the downward trend is more pronounced. In the early stages of training (rounds 0–50), the migration cost of both algorithms is high, with DDPG's initial cost being approximately 10 and HADDPG's approximately 8.5. This indicates that the system is not fully optimized in the initial stages, resulting in high resource migration consumption. However, HADDPG's migration cost decreases more quickly, converging to a lower cost level more quickly than DDPG, indicating that the algorithm is able to identify more optimal strategies earlier and reduce resource waste. 2) The overall migration cost is lower. In the middle and late stages of training (rounds 50–500), the migration cost of the HADDPG algorithm is significantly lower than that of DDPG.

[0140] HADDPG: Migration cost is stable at around 2.5–3.0. DDPG: Migration cost is stable at around 3.5–4.0. This difference indicates that HADDPG demonstrates greater efficiency in optimizing task scheduling and resource allocation, reducing overall resource migration consumption. In terms of curve fluctuations, HADDPG's migration cost curve is smoother and has smaller fluctuations. In contrast, DDPG's curve fluctuates more, especially in the mid-term stage (100–300 rounds), where significant repetitive fluctuations occur. This indicates that its migration strategy optimization is still unstable and has failed to continuously reduce costs.

[0141] Figure 7 This figure shows the changing trends of the total loss (Total Loss) of the HADDPG and DDPG algorithms over 500 training episodes. The vertical axis represents the total loss, and the horizontal axis represents the number of training episodes. HADDPG curve (yellow): The total loss is high in the initial stage, but as training progresses, the loss decreases rapidly and gradually stabilizes. After 100 episodes, the fluctuation is small, indicating good convergence and stability. DDPG curve (red): The total loss is also high in the initial stage and decreases quickly, but the overall loss is high and fluctuates greatly, especially in the later stages of training (after 300 episodes), when the fluctuation is more obvious, and the convergence effect is not as good as HADDPG.

[0142] Figure 8The figure shows the changes in user service latency for the DDPG and HADDPG algorithms over 500 training rounds. The vertical axis represents latency, and the horizontal axis represents the number of training rounds. The DDPG curve (solid line) shows high overall latency and large fluctuations, with significant peaks during training, reaching a maximum latency of nearly 0.24. This indicates that the DDPG algorithm is less stable in task processing and exhibits significant latency fluctuations. The HADDPG curve (dashed line) shows significantly lower latency than DDPG, with less fluctuation and more stable performance, with an average latency of approximately 0.15. HADDPG maintains consistently low latency throughout the training rounds, demonstrating superior optimization results and robustness.

[0143] In summary, the satellite-ground collaborative service migration method designed by the present invention in satellite edge computing has been implemented, and the effectiveness of information extraction and the superiority of the algorithm structure of multiple actor critics have been verified.

Claims

1. A satellite-ground collaborative service migration method in satellite edge computing, characterized in that: An information extraction structure is constructed to extract key timing information. A multi-agent reinforcement learning algorithm for satellite-ground collaborative service migration is designed. The key timing information is used as observations in the multi-agent reinforcement learning algorithm for satellite-ground collaborative service migration. Reinforcement learning is used to obtain the optimal service migration strategy. The specific processing process of the information extraction structure is as follows: Input sequence , contains the state information of each time step and action information , the input at each time step ; In the service migration scenario, status information Including the current user's location, server status information, bandwidth, user access delay, action information Represents the action performed by the server at that moment; First, a feedforward neural network is used to train the input sequence at each time step The attention score is obtained by processing. The specific calculation formula is: (1)； in, is the weight vector, is the weight matrix, is the bias vector; Converting attention scores into weights: Attention scores Converted into attention weight through softmax function , ensuring that the weights sum to 1, so that each time step contributes a proportion to the total migration decision: (2)； Weighted calculation of temporal attention representation: by attention weight Perform weighted summation on the input sequence to obtain the temporal attention representation : (3)； The forward gated recurrent unit processes data from time steps 1 to T and generates the forward hidden state : (4)； The backward gated recurrent unit processes the data from time step T to 1 and generates the backward hidden state : (5)； The bidirectional gated recurrent unit uses three main gating mechanisms: update gate , Reset Gate and candidate hidden states , these gating mechanisms are used to control the flow of information and state updates; the specific calculation process is as follows: (6)； is the Sigmoid activation function, and is the weight matrix, is the current input, is the hidden state at the previous moment, Is the bias term; the update gate controls the balance between the previous state and the current candidate state; (7)； (8)； (9)； The forward hidden state and the backward hidden state are concatenated together to form a bidirectional hidden state: (10)； Hidden states serve as observation values ​​for agents in multi-agent reinforcement learning algorithms for service transfer in satellite-ground collaboration.

2. The satellite-ground collaborative service migration method in satellite edge computing according to claim 1, characterized in that: The information extraction structure includes a temporal attention mechanism and a bidirectional gated recurrent unit to extract temporal patterns in historical data. By introducing attention to key time steps and combining the bidirectional gated recurrent unit to capture bidirectional temporal dependencies, the predictive ability of temporal data is improved. The temporal attention mechanism calculates the attention weights of the state and action information at each time step. The bidirectional gated recurrent unit processes data simultaneously from both the past and future directions, capturing long-term and short-term dependencies and trends in historical data.

3. The satellite-ground collaborative service migration method in satellite edge computing according to claim 2, characterized in that: The temporal attention mechanism operates as follows: a) Critical moment identification and attention weight allocation: During service migration, the temporal attention mechanism calculates the weights of the state and action information at each time step; b) Dynamic weight adjustment and adaptive migration decision-making: The temporal attention mechanism dynamically adjusts weights based on real-time status. When a user is about to leave the current satellite coverage area, the attention mechanism assigns a higher weight; The temporal attention mechanism selectively focuses on the most critical parts of historical data, captures state correlations in events, and constructs a temporal view that highlights relationships and patterns across time.

4. The satellite-ground collaborative service migration method in satellite edge computing according to claim 3, characterized in that: The bidirectional gated recurrent unit includes a forward gated recurrent unit and a backward gated recurrent unit, which performs bidirectional information capture and comprehensive hidden state representation; The temporal view performs bidirectional processing through a bidirectional gated recurrent unit, simultaneously considering the past and future dependencies of the data, capturing hidden trends and patterns, and effectively integrating temporal information with sequence dependencies. During the service migration process, the forward gated recurrent unit predicts the current state based on the user's past state information and network link changes, and extracts the user's state information at the future time; The backward gated recurrent unit starts processing information forward at the last time step of the time series to capture the potential impact of future time steps on the current time step; The backward gated recurrent unit predicts future network topology changes and adjusts service migration decisions in advance, reducing service interruption time and migration delay; By concatenating the hidden states of the forward gated recurrent unit and the backward gated recurrent unit, a comprehensive hidden state representation is obtained. , as the observed value.

5. The satellite-ground collaborative service migration method in satellite edge computing according to claim 1, characterized in that: The satellite-ground collaborative service migration multi-agent reinforcement learning algorithm includes two local actor-critic structures and a global critic network; the two local actor-critic structures are respectively the ground base station actor-critic structure and the satellite network actor-critic structure; the local actor-critic structures are used to process the local service migration decision of the ground base station and the local service migration decision of the satellite network; The ground base station and satellite network are heterogeneous in terms of status information; the status information of the ground base station includes the load, bandwidth, computing power, user location, and service requirements of the local network; the status information of the satellite network includes the satellite's orbit information, signal quality, coverage, satellite-to-ground link bandwidth, and satellite available computing resources; The global critic network is used to evaluate the overall effect of local service migration decisions and pass the evaluation results to each local actor-critic network.

6. The satellite-ground collaborative service migration method in satellite edge computing according to claim 5, characterized in that: The multi-agent reinforcement learning algorithm for service migration in satellite-ground collaboration is specifically as follows: In each actor-critic structure, agents update their strategies based on local state and action information to achieve local optimal decisions. By introducing a global critic network, all agents' decisions are globally evaluated and coordinated. This global critic network comprehensively evaluates the behavior of all agents and provides global feedback, ensuring that each agent can achieve local optimization while also taking into account global network performance goals. In the local Actor-Critic architecture for ground base stations, each agent represents a different ground base station and is used to manage service migration decisions in the base station network, taking into account the base station's computing resources, storage resources, bandwidth, and user needs. Actors are used to generate migration actions for ground base stations, and Local Critic is used to evaluate ground base station migration decisions. When a user moves from the coverage area of ​​one ground base station to the coverage area of ​​another, the possibility of triggering service migration of the base station is determined. In the local Actor-Critic architecture of satellite networks, it is used to process migration decisions in satellite networks, taking into account satellite orbits, inter-satellite transmission capabilities, and limited computing and storage resources. When the satellite to which a user is connected leaves the coverage area, service migration between satellites is triggered. The global critic network provides global feedback on migration decisions between ground base stations and satellite networks; For ground base stations, the global critic network evaluates the contribution of the ground base station's service migration decision to the entire network, including service interruption time and latency. For satellite networks, the global critic network evaluates the impact of the satellite network on latency, link quality, and resource consumption during service migration. The global critic network performs global collaborative optimization: the global critic network evaluates the effect of each local decision and provides feedback to each actor based on the actual network status.

7. The satellite-ground collaborative service migration method in satellite edge computing according to claim 6, characterized in that: Based on local and global reward values, the service delay and energy consumption of both the ground base station and the satellite network are evaluated, and their respective local reward values ​​are calculated. A global critic architecture is then used to coordinate the behaviors of the two to minimize the overall user service delay. a) Calculation of local reward values ​​for ground base stations: When calculating the local reward values ​​for ground base stations, service latency and energy consumption are taken into account. The local reward values ​​of ground base stations are calculated based on the service latency and energy consumption of the user equipment (UE) they serve. The lower the service latency and energy consumption, the higher the reward value of the ground base station. The calculation formula is: (11); in, represents the service delay of the user equipment served by the ground base station, represents the energy consumption of the user equipment served by the ground base station; The calculation delay is , communication delay and migration delay Composition, calculated as follows: (12); in is the energy consumed by the ground base station calculation , the energy consumed by communicating with ground users and the energy consumed by service migration between ground base stations Composition, calculated as follows: (13)； b) Calculation of the local reward value of the satellite network: When calculating the local reward value of the satellite network, two key factors are considered: service latency and energy consumption. The local reward value of the satellite network is evaluated not only based on the service latency of the user equipment (UE) it serves, with shorter latency resulting in higher rewards, but also considering the satellite's energy consumption, with lower energy consumption resulting in higher rewards. (14)； in, represents the service delay of the satellite network server, represents the energy consumption of users of satellite services; is the energy consumed by satellite network computing , the energy consumed by communicating with ground users and the energy consumed by service migration between satellite networks Composition, calculated as follows: (15)； c) Reward calculation of the global critic structure: ； (16)； in, represents the service delay of the ground base station, represents the energy consumption of the ground base station, represents the service delay of the satellite network server, Represents the energy consumption of users served by satellite network.

Images