Low earth orbit satellite constellation network edge computing multi-level offloading method based on reinforcement learning

Abstract

The application discloses a low-orbit satellite constellation network edge computing multi-stage unloading method based on reinforcement learning and belongs to the technical field of satellite communication networks and edge computing technology. A traditional edge computing unloading method is designed for a ground network and does not consider the characteristics of a low-orbit satellite constellation network. The application introduces the edge computing technology into the low-orbit satellite constellation network, directly executes various edge computing tasks with strong timeliness on the low-orbit satellite, and can effectively reduce the edge computing task delay, improve the system resource utilization and improve the user experience quality.

Description

Technical Field

[0001] This invention belongs to the field of satellite communication network and edge computing technology, and in particular relates to a multi-level offloading method for edge computing tasks in low-Earth orbit satellite constellation networks based on multi-agent reinforcement learning. Background Technology

[0002] In recent years, with the rapid advancement of space and information technology, the industry has attached great importance to the construction of a new generation of low-Earth orbit (LEO) satellite constellation networks. In the future construction of 6G networks, this new generation of LEO satellite constellation networks will be an important supplement to terrestrial networks. Edge computing technology can effectively reduce latency in edge computing tasks, lower network bandwidth consumption, and protect user data security. Since its inception, it has developed rapidly and has been widely applied in fields such as the Internet of Things (IoT), telecommunications broadband networks, and the industrial internet, and is undergoing increasingly in-depth application research and practical exploration. Applying edge computing technology to LEO satellite constellation networks with large spatiotemporal scales can bring more significant network performance gains, thus becoming one of the current research hotspots for LEO satellite constellation networks. LEO satellite constellation networks have characteristics that distinguish them from traditional terrestrial broadband networks, such as large spatiotemporal scales, highly dynamic topology changes, and strong resource heterogeneity. Their edge computing theoretical research faces new challenges, including a lack of system architecture and model design, difficulty in improving the utilization rate of multi-dimensional heterogeneous resources, and difficulty in guaranteeing latency for deterministic task services.

[0003] Edge computing offloading refers to the process by which resource-constrained devices completely or partially offload computationally intensive tasks to resource-rich network edge nodes. This addresses the resource and energy efficiency limitations of end devices, reduces pressure on the core network, and lowers latency caused by transmission. However, most existing typical edge computing offloading algorithms are developed for terrestrial network applications and do not consider the characteristics of low-Earth orbit (LEO) satellite constellation network nodes, such as high-speed movement and dynamic topology changes. Consequently, they fail to achieve efficient utilization of LEO satellite onboard computing resources to further improve the service quality of LEO satellite constellation networks. Summary of the Invention

[0004] Introducing edge computing technology into low-Earth orbit (LEO) satellite constellation networks allows for the direct execution of time-sensitive edge computing tasks on LEO satellites, effectively reducing latency, improving system resource utilization, and enhancing user experience. However, traditional edge computing offloading methods are designed for terrestrial networks and do not consider the characteristics of LEO satellite constellation networks. Therefore, this invention proposes a multi-level offloading method for edge computing tasks in LEO satellite constellation networks based on Multi-Agent Deep Reinforcement Learning (MADRL) to reduce latency and improve the utilization of onboard computing resources.

[0005] The technical solution adopted in this invention is as follows:

[0006] 1: Determine the multi-level offloading strategy for edge computing tasks in low-Earth orbit satellite constellation networks.

[0007] In low Earth orbit (LEO) satellite constellation networks, satellites move at high speeds along near-Earth orbits, forming a network covering the Earth's surface to provide seamless wireless broadband services to terrestrial users. Typically, terrestrial users dynamically select the nearest LEO satellite as their network access node; current research and technologies usually treat the access satellite as the sole edge computing node and the ground-based cloud computing center as the next-hop computing node. This invention innovatively proposes a multi-level edge computing offloading model, utilizing inter-satellite links to achieve collaborative task offloading across multiple LEO satellites, thereby reducing task latency and improving the utilization rate of onboard computing resources. More specifically, terrestrial users directly send edge computing tasks to their access satellites, which then execute edge computing task offloading decisions based on local observation information. Each LEO satellite can choose to execute three different types of offloading decisions:

[0008] 1) Allocate local edge computing resources to execute the task. After the task is completed, return the calculation results to the user along the task upload path.

[0009] 2) Forward the task to one of four neighboring satellites orbiting in the same or adjacent orbits;

[0010] 3) The task is sent directly to the ground cloud computing center behind the nearest gateway station, and the task is processed by the computing resources of the cloud computing center. The calculation results of the task are returned to the user via satellite.

[0011] If a low-Earth orbit (LEO) satellite decides to forward the mission to one of its neighboring LEO satellites, the above process will be repeated on the next LEO satellite node.

[0012] 2. Construct a multi-level offloading model for edge computing tasks in a low-Earth orbit satellite constellation network.

[0013] This invention will construct a ground user model, an edge computing task model, a task latency model, and a resource utilization model for a low-Earth orbit satellite constellation network, and on this basis, model the system performance optimization problem.

[0014] 2.1 User Model

[0015] The distribution of users for edge computing services in low-Earth orbit satellite constellations is uneven. For example, satellites over densely populated urban areas experience high computing loads, while satellites over remote areas such as deserts and mountains have largely idle computing resources. This invention divides the ground into M regions with uneven user distribution, where user locations are evenly distributed within each region. In this invention, a probabilistic statistical model is used to characterize user edge computing requests over a period of time. It is assumed that all user actions are independent and periodically send edge computing task requests to the access satellites, and the time interval between two adjacent tasks for a single user follows a negative exponential distribution, with the probability density function being:

[0016]

[0017] 2.2 Edge Computing Task Model

[0018] Edge computing in low-Earth orbit satellite constellation networks is generally suitable for time-sensitive tasks with low computational requirements. Based on the time sequence of user-generated tasks, this invention uses a time sequence {Task1, Task2, ..., Task...} total} to represent all edge computing tasks, and to assign any task to Task i Modeling as Where S i L represents the size of the data for this task. i This is the maximum unloading time for this task, U i Points to the user who generated the task. This indicates the time point at which the edge computing task was generated. This invention assumes that all tasks are independent of each other, and that the data volume of each task follows a truncated normal distribution with an expectation of μ and a variance of σ. 2 .

[0019] 2.3 Task Latency and Resource Utilization Model

[0020] First, the task of collaboratively offloading edge computing tasks will be participated in. i All satellites are modeled as And any one of the LEO satellites i j The unloading decision model is This indicates that the task is processed locally on the satellite. i , and This indicates that the mission will be forwarded to four neighboring satellites. This indicates that the task will be sent to the ground cloud computing center.

[0021] (1) Delay Model

[0022] Reducing edge computing task latency is a key optimization objective of this invention. The latency considered in this invention mainly includes task upload latency, task queuing latency, task computation latency, and task return latency, i.e.

[0023] Task latency = Task upload latency + Task queuing latency + Task calculation latency + Task return latency

[0024] The data transmission rates of satellite-to-ground communication links and inter-satellite communication links are respectively represented by v. gs and v ss This indicates that the task is located at the edge computing level. i During the upload process, the latency caused by satellite-to-ground data transmission is expressed as... Inter-satellite data transmission latency is Among them l gs and l ss These are the average distances for satellite-to-ground and inter-satellite transmission, respectively, and c represents the speed of light. Therefore, Task i The total upload latency can be calculated as follows:

[0025]

[0026] like This indicates that the task has been sent to the cloud computing center for processing. This indicates that the task was executed on a satellite node. 'd' represents the number of inter-satellite transfers involved in the task's unloading process. Task i The formula for calculating the computational delay is:

[0027]

[0028] Where ξ is the CPU computation density, f LEO and f Cloud These represent the computing resources allocated to this task by the satellite and the cloud computing center, respectively. If Task i If a task is processed onboard but onboard computing resources are not available, it will be placed in a queue and will not be processed until resources become available onboard, introducing a queuing delay D. i,q This invention assumes that the cloud computing center has sufficient computing resources, and that if tasks are sent to the cloud computing center for processing, congestion and queuing will not occur. If Task i Unable to reach the time limit L i Once the computation is complete, the computing resources allocated to this task will be released, and the task is considered to have failed to be unloaded; if Task i In L i The data was successfully processed, and the data size was [data missing]. The task results will be returned to the ground user. The formula for calculating the result return delay is as follows:

[0029]

[0030] Based on the above modeling process, the edge computing task can be obtained. i The total processing latency is:

[0031] D i =D i,up +D i,q +D i,C +D i,down

[0032] (2) Calculation of resource utilization rate model

[0033] Because of the uneven distribution of users in low-Earth orbit (LEO) satellite networks, the computational load on LEO satellites is extremely uneven, leading to inefficient use of onboard computing resources. Therefore, improving the utilization rate of onboard edge computing resources is another important optimization objective of this invention. At time t i For Task i After making the offloading decision, any LEO satellite in the low Earth orbit constellation k The onboard computing resource utilization rate (CRUR) can be expressed as:

[0034]

[0035] Among them, CN k It is the current LEO k The number of edge computing tasks being processed, f LEO and These represent the computing resources allocated to a satellite for a single mission and the total onboard computing resources, respectively. Therefore, the overall resource utilization rate of the entire low-Earth orbit satellite constellation network is:

[0036]

[0037] Where numSat represents the total number of satellites in the constellation.

[0038] 2.4 Modeling the Performance Optimization Problem

[0039] This invention proposes a multi-level offloading model for edge computing tasks in low-Earth orbit satellite constellation networks. The optimization objectives are to reduce task latency and improve the utilization rate of onboard computing resources. Therefore, the cost function of the optimization problem is modeled as Cost. i =φ1D i +φ2(1-CRUR(t i ), where φ1 and φ2 are weights used to balance different optimization objectives. Based on the cost function, the optimization problem of this invention can be further formulated as:

[0040]

[0041] st

[0042] C1:

[0043] C2:

[0044] C3:

[0045] C4:D i,up +D i,q +D i,C ≤L i

[0046] C5:φ1+φ2=1

[0047] In this equation, C1, C2, C3, C4, and C5 are constraints to ensure the effectiveness of the task unloading decision: C1 restricts the task unloading decision variables to binary numbers, C2 represents... The optional offloading decision is only two options: process the task locally or send the task to a ground cloud computing center for processing. C3 explicitly defines the Task. i In addition to the upload path Other low-Earth orbit satellites only have four optional offloading strategies: forwarding the task to a neighboring satellite. C4 requires all edge computing tasks to be completed within the offloading time limit; otherwise, the task offloading is considered a failure, and all computing resources are released.

[0048] 3. Transform the performance optimization problem into a locally observable Markov decision process (POMDP).

[0049] In low-Earth orbit (LEO) satellite constellation networks with large spatial scale characteristics, centralized control nodes struggle to acquire the global network state in a timely manner for real-time offloading decisions. Therefore, this invention defines satellites as independent intelligent agents that determine the next-hop offloading decision based solely on local observation information. Consequently, this invention transforms the offloading problem of edge computing tasks in LEO satellite constellation networks into a locally observable Markov decision process, which can be described by the following 6-element tuple:

[0050] P = (S, A, T, R, O, γ)

[0051] Where S = {s1, s2, ..., s} n} is the global state space of the environment, A = {a1, a2, ..., a} n} is the set of actions shared by the agents, T = T(S) t+1 =s′|S t =s,A t=a) is the state transition function of the environment, R = S × A is the global reward function shared by the agents, O = {o1, o2, ..., o n Let} represent the local observation state space of the agent, and γ∈[0,1] is a discount factor used to balance short-term and long-term rewards. In the scenario focused on by this invention, the actions and reward functions for the local observation state are more specifically defined as follows:

[0052] Local Observable State: In a low-Earth orbit (LEO) satellite constellation network, each LEO satellite acts as an agent, and its local observation state space is defined as O(t) = {f}. obs ,task,position}. Where It represents the idle status of computing resources on the satellite and its neighboring satellites. task={size,ub,hop} indicates important information about the task to be unloaded, including the size of the task data, the upper limit of the task processing time, and the number of hops that the task has been unloaded. Finally, position points to the location of the low-Earth orbit satellite.

[0053] Action: Each agent receives any edge computing task. i After that, they will all be in the action space Choose one action to perform an unload decision on the task. Possible actions include local processing of the task. i , Task i Forward to neighboring satellites for processing, and assign tasks i It is sent to the ground cloud computing center for processing.

[0054] Reward Function: The goal of each agent is to learn the optimal offloading strategy to reduce offloading costs, i.e., reduce task latency and improve onboard resource utilization. Therefore, the reward function can be defined as a weighted sum of two metrics. When the agent makes a better offloading decision, it receives a more positive reward. Thus, the agent is called a Task. i The immediate reward for making the decision to uninstall the task can be set as follows: Among them ΔCRUR i express After making the offloading decision, the relative rate of change in resource utilization can be calculated using the following formula:

[0055]

[0056] in and They are The network resource utilization before and after making the offloading decision. To consider the long-term impact of the agent's decision, the long-term cumulative reward of the action should be further calculated. Here, γ is the decay factor, which determines the impact of the current unloading decision on future rewards. The smaller γ is, the higher the proportion of immediate rewards.

[0057] 4. A multi-level task offloading algorithm based on Multi-Agent Deep Reinforcement Learning (MADRL) is proposed.

[0058] To ensure that LEO satellites learn the optimal offloading strategy, this invention proposes a task offloading algorithm based on Multi-Agent Reinforcement Learning (MADRL). This algorithm applies the Actor-Critic framework and is implemented through centralized training and distributed execution. On one hand, the Actor, or policy network, is deployed on each LEO satellite. It learns a policy μ to output the probability distribution of selectable actions based on local observations and then executes the action with the highest probability. Since LEO satellites in the same constellation have similar altitudes, the policy networks of all satellites can share parameter values. On the other hand, the Critic, or evaluation network, is deployed on the central control node and is responsible for estimating the action-value function Q based on global state information to better evaluate the agent's actions. The Actor updates its policy based on the Critic's evaluation. Furthermore, the centralized Critic needs to acquire global network state information for evaluating action values, while the distributed Actors must synchronously update their neural network parameters. Therefore, the communication mechanism between Actors and the Critic is crucial for centralized model training. The Critic and communication mechanism are designed for the training process; after the model is trained, they do not need to be deployed in the actual application scenario, thus not affecting the real-time performance of decision-making.

[0059] Since edge computing tasks are randomly generated by the user, the offloading strategy implemented by LEO satellites is described as a task-driven asynchronous action. The MADRL-based task offloading algorithm assumes that all LEO satellites are isomorphic and fully cooperative agents, sharing a policy μ and its parameters θ. Agents continuously interact with the environment to generate experiences {s,s′,o,o′,a,r} and store them in an experience pool. During training, the neural network is trained by repeatedly drawing small batches of training data randomly from the experience pool to update the parameter values ​​of Actors and Critic. This invention uses the deterministic policy gradient (DPG) algorithm to update the policy network (Actors):

[0060]

[0061]

[0062] Where s represents the global state information of the satellite network, and a represents the action performed by the Actor based on local observation state variables. The centralized value network, Critic, has a neural network parameter ν. Critic takes global state information and agent actions as input and outputs the Q-value for each state-action combination. In each multi-step update iteration, this invention employs the target value network Q... ν′ and target policy network μ θ′ To fix a target value y:

[0063] y = r + γQ ν' (s′,μ θ′ (o′)),

[0064] Where r is the immediate reward of the state-action combination (s, a), and (s′, a′) is the action-state combination at the next moment. To alleviate the overestimation problem of the evaluation network, this invention utilizes two evaluation networks Q. v1 and Q v2 To estimate the same set of training samples in parallel, the smaller estimate is selected as the final estimate. Therefore, the objective of evaluating the network update is modified as follows:

[0065]

[0066] Q v The Loss function of (s,a) can be expressed as the following formula:

[0067]

[0068] With the goal of minimizing the loss function value, gradient descent and backpropagation are used to continuously optimize the parameters of the value network in each iteration, enabling the value network to more accurately evaluate the agent's policy.

[0069] This invention introduces edge computing technology into low-Earth orbit satellite constellation networks, enabling the direct execution of various time-sensitive edge computing tasks on low-Earth orbit satellites. This can effectively reduce the latency of edge computing tasks, improve system resource utilization, and enhance the quality of user experience. Attached Figure Description

[0070] Figure 1 This is a diagram of the multi-level offloading architecture for edge computing in a low-Earth orbit satellite constellation network in an example of the present invention.

[0071] Figure 2 This is a flowchart of the Multi-Agent Reinforcement Learning (MADRL) algorithm in an example of the present invention;

[0072] Figure 3 This refers to the average offloading latency of edge computing tasks in this invention example;

[0073] Figure 4 This refers to the utilization rate of spaceborne edge computing resources in this invention example;

[0074] Figure 5 This represents the average offloading cost of edge computing tasks in this invention example. Detailed Implementation

[0075] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and specific examples. The following specific examples are for illustrative purposes only and are not intended to limit the scope of the invention. The specific implementation of the present invention is divided into two stages: Stage 1 involves training a deep reinforcement learning model through continuous interaction between the agent and the environment; Stage 2 involves deploying the trained deep reinforcement learning model in a real system to perform an unloading task.

[0076] Phase 1: Model Training Phase

[0077] Step 1: Build a user model

[0078] First, an edge computing user model is constructed based on a real ground population density distribution dataset, and an edge computing task sequence is generated based on the user behavior model. The user distribution of edge computing services in the low-Earth orbit satellite constellation network is extremely uneven. For example, satellites above densely populated urban areas have high computing loads, while satellites above remote areas such as deserts and mountains have most of their computing resources idle. Referring to NASA's open-source real ground population distribution dataset, this invention divides the ground into M regions with uneven user distribution, where user locations are evenly distributed within each region. In this invention, a probabilistic statistical model is used to characterize user edge computing requests over a period of time. It is assumed that all user behaviors are independent and periodically send edge computing task requests to the access satellites, and the time interval between two adjacent tasks for a single user follows a negative exponential distribution, with the probability density function being:

[0079]

[0080] in, The expected time interval for sending requests to a single user;

[0081] Step 2: Generate an edge computing task sequence

[0082] Edge computing in low-Earth orbit satellite constellation networks is generally suitable for time-sensitive tasks with low computational requirements. Based on the time sequence of user-generated tasks, this invention uses a time sequence {Task1, Task2, ..., Task...} total} to represent all edge computing tasks, and to assign any task to Task i Modeling as Where S i L represents the size of the data for this task.i This is the maximum unloading time for this task, U i Points to the user who generated the task. This indicates the time point in time when the edge computing task was generated. This invention assumes that all tasks are independent of each other, and that the data volume of each task follows a truncated normal distribution with an expectation of μ and a variance of σ. 2 .

[0083] Step 3: Construct a dynamic topology model for the low-Earth orbit satellite constellation network

[0084] This invention uses the classic Iridium constellation as the target network, which contains N satellites. orbit Tracks, N orbit =6, each orbital has N Sat_orbit N uniformly distributed low-orbit satellites Sat_orbit =11. The satellite moves at high speed along its orbit, providing seamless wireless broadband service to ground users. Typically, ground users will dynamically select the nearest low-Earth orbit satellite as their access node. This network topology is symmetrical about the line connecting the two poles, with a longitude difference β of 31.6° between orbits in the same direction and an orbital altitude h. orbit The range is 780 km. The specific parameter values ​​for the Iridium network topology are shown in Table 1.

[0085] Table 1 Iridium network topology parameter values

[0086] Number of orbits <![CDATA[N orbit ]]> 6 Number of satellites per orbit N Sat_orbit = 11 11 Orbit altitude h orbit ]]> 780 km Satellite movement speed v sat ]]> 7.46 km / s Longitude difference between prograde orbits β 31.6° Longitude difference between retrograde orbits α 22° Orbit semi-major axis r a ]]> 7 185 km Orbit eccentricity e 0 Perigee angle of inclination ω 0° Orbit inclination angle i 86.4°

[0087] Step 4: After receiving the edge computing task, the satellite obtains the local observable state.

[0088] In low-Earth orbit (LEO) satellite constellation networks with large spatial scale characteristics, centralized control nodes struggle to acquire the global network state in a timely manner for real-time unloading decisions. Therefore, this invention defines satellites as independent intelligent agents, determining the next-hop unloading decision based solely on locally observable state information. Each LEO satellite is considered an intelligent agent, and its locally observable state space is defined as O(t) = {f obs ,task,position}. Where It represents the idle status of computing resources on the satellite and its neighboring satellites. task={size,ub,hop} indicates important information about the task to be unloaded, including the size of the task data, the upper limit of the task processing time, and the number of hops that the task has been unloaded. Finally, position points to the location of the low-Earth orbit satellite.

[0089] Step 5: The satellite makes an offloading decision based on the local observable state.

[0090] Each agent receives an edge computing task. iAnd after observing local information, it will all be from the action space Choose one action to perform an unload decision on the task. Possible actions include local processing of the task. i , Task i Forward to neighboring satellites for processing, and assign tasks i The data is sent to a ground-based cloud computing center for processing. This invention employs an Actor-Critic framework, where an Actor, or policy network, is deployed on each LEO satellite. By learning a policy μ, it outputs a probability distribution of possible actions based on local observations and then executes the action with the highest probability. Because LEO satellites within the same constellation have similar altitudes, the policy networks of all satellites can share parameter values.

[0091] During the training phase, the agent's choice of unloading strategy is divided into two cases: exploration and exploitation. This invention uses the ε-greedy algorithm to make a trade-off between exploration and exploitation based on probability: the agent tries to explore randomly with a probability of ε and exploits the current optimal strategy with a probability of 1-ε, which can collect training samples more widely to a certain extent.

[0092] Step 6: Update the edge computing task sequence

[0093] If the current task is forwarded to a neighboring satellite, then according to the time when the current task arrives at the neighboring satellite, insert the task back into the edge computing task sequence generated in step 1 in chronological order, and proceed to step 3; if the current task is executed, then remove the task from the edge computing task sequence and proceed to step 7.

[0094] Step 7: Calculate the reward value for the offloading decision made by the satellite.

[0095] If an edge computing task is executed, the reward value for the multi-level offloading decision will be calculated after the task result is returned to the user. This reward value will then be sent to all satellites participating in the offloading task, generating training samples stored in the Replay Buffer. Each agent aims to learn the optimal offloading strategy to reduce offloading costs, i.e., reduce task processing and improve onboard resource utilization. Therefore, the total latency of the task and the network's edge computing resource utilization need to be obtained before calculating the reward value. First, the edge computing tasks participating in the collaborative offloading will be... i All satellites are modeled as and any one of the satellites The unloading decision model is This indicates that the task is processed locally on the satellite. i , and This indicates that the mission will be forwarded to four neighboring satellites. This indicates that the task will be sent to the ground cloud computing center.

[0096] Step 7.1 Calculate task latency

[0097] Reducing the processing latency of edge computing tasks is a key optimization objective of this invention. The latency considered in this invention mainly includes task upload latency, task queuing latency, task computation latency, and task data return latency. The data transmission rates of the satellite-to-ground communication link and the inter-satellite communication link are respectively determined by v gs and v ss This indicates that the task is located at the edge computing level. i During the upload process, the latency caused by satellite-to-ground data transmission is... The latency caused by inter-satellite data transmission is Among them l gs and l ss These are the average distances for satellite-to-ground and inter-satellite transmission, respectively, and c represents the speed of light. Therefore, Task i The total upload latency can be calculated as follows:

[0098]

[0099] like This indicates that the task has been sent to the cloud computing center for processing. This indicates that the task was executed on a satellite node. 'd' represents the number of inter-satellite transfers involved in the task's unloading process. Task i The formula for calculating the computational delay is:

[0100]

[0101] Where ξ is the CPU computation density, f LEO and f Cloud These represent the computing resources allocated to this task by the satellite and the cloud computing center, respectively. If Task i If the task is processed onboard but onboard computing resources are not yet available, it will be placed in a queue and will not be processed until resources onboard are released, introducing a queuing delay D. i,q This invention assumes that the cloud computing center has sufficient computing resources, and that if tasks are sent to the cloud computing center for processing, congestion and queuing will not occur. If Task i Unable to reach the time limit L i Once the computation is complete, the computing resources allocated to this task will be released, and the task is considered to have failed to unload; if Task i In L i The data was successfully processed, and the data size was [data missing]. The task results will be returned to the ground user. The formula for calculating the result return latency is as follows:

[0102]

[0103] Based on the above modeling process, the edge computing task can be obtained. i The total processing latency is:

[0104] D i =D i,up +D i,q +D i,C +D i,down

[0105] Step 7.2 Calculate the onboard resource utilization rate

[0106] Because the users served by low-Earth orbit (LEO) satellite constellation networks are unevenly distributed, the computational load on LEO satellites is extremely uneven, leading to inefficient use of onboard computing resources. Therefore, improving the utilization rate of onboard edge computing resources is another important optimization objective of this invention. At time t i For Task i After making the offloading decision, any LEO satellite in the low Earth orbit constellation k It can be represented as:

[0107]

[0108] Among them, CN k It is the current LEO k The number of edge computing tasks being processed, f LEO and These represent the computing resources allocated to a satellite for a single mission and the total onboard computing resources, respectively. Therefore, the overall resource utilization rate of the entire low-Earth orbit satellite constellation network is:

[0109]

[0110] Where numSat represents the total number of satellites in the constellation.

[0111] Step 7.3 Calculate the unloading decision reward value

[0112] Then, based on the values ​​of two indicators—task latency and onboard resource utilization—the reward function is defined as the weighted sum of the two indicators. When the agent makes a better offloading decision, it receives a more positive reward. Therefore, the agent is Task. i The immediate reward for making the decision to uninstall the task can be set as follows: Among them ΔCRUR i express After making the offloading decision, the relative rate of change in resource utilization can be calculated using the following formula:

[0113]

[0114] in and They are The network resource utilization before and after making the offloading decision. To consider the long-term impact of the agent's decision, the long-term cumulative reward of the action should be further calculated. Here, γ is the decay factor, which determines the impact of the current unloading decision on future rewards. The smaller γ is, the higher the proportion of immediate rewards.

[0115] Step 8: Train a multi-agent deep reinforcement learning model

[0116] A minibatch is randomly selected from the Replay Buffer to train the actor policy network and the centralized critic value network. This invention uses the deterministic policy gradient (DPG) algorithm to update the policy network (Actor):

[0117]

[0118]

[0119] Where s represents the global state information of the satellite network, and a represents the action performed by the Actor based on local observation state variables. The centralized value network, Critic, has a neural network parameter ν. Critic takes global state information and agent actions as input and outputs the Q-value for each state-action combination. In each multi-step update iteration, this invention employs the target value network Q... ν′ and target policy network μ θ′ To fix a target value y:

[0120] y = r + γQ ν' (s′,μ θ′ (o′)),

[0121] Where r is the immediate reward of the state-action combination (s, a), and (s′, a′) is the action-state combination at the next moment. To alleviate the overestimation problem of the evaluation network, this invention utilizes two evaluation networks Q. v1 and Q v2 To estimate the same set of training samples in parallel, the smaller estimate is selected as the final estimate. Therefore, the objective of evaluating the network update is modified as follows:

[0122]

[0123] Q v The Loss function of (s,a) can be expressed as the following formula:

[0124]

[0125] With the goal of minimizing the loss function value, gradient descent and backpropagation are used to continuously optimize the parameters of the value network in each iteration, enabling the value network to more accurately evaluate the agent's policy.

[0126] Phase Two: Model Execution Phase

[0127] Step 1: Deploy a deep reinforcement learning model

[0128] The Actors (policy networks) of the trained deep reinforcement learning model are deployed on each low-Earth orbit satellite within the Iridium constellation. The centralized Critic (evaluation network) does not participate in the model execution phase.

[0129] Step 2: Satellite acquires local observable state information

[0130] The initialization task sequence is performed, and edge computing tasks are sent to the access satellite. The satellite acquires local state information observations. Each low-Earth orbit satellite acts as an intelligent agent, and its local observation state space is defined as O(t) = {f}. obs ,task,position}. Where It represents the idle status of computing resources on the satellite and its neighboring satellites. task={size,ub,hop} indicates important information about the task to be unloaded, including the size of the task data, the upper limit of the task processing time, and the number of hops that the task has been unloaded. Finally, position points to the location of the low-Earth orbit satellite.

[0131] Step 3: The satellite makes an offloading decision based on the local observable state.

[0132] Each agent receives an edge computing task. i And after observing local information, it will all be from the action space Choose one action to perform an unload decision on the task. Possible actions include local processing of the task. i , Task i Forward to neighboring satellites for processing, and assign tasks i The data is sent to a ground-based cloud computing center for processing. This invention employs the Actor-Critic framework, where the Actor, or policy network, is deployed on each low-Earth orbit satellite. It learns a policy μ to output a probability distribution of possible actions based on local observations, and then executes the action with the highest probability. Unlike the training phase, the execution phase does not employ an ε-greedy policy to explore the boundary; the agent only selects the optimal policy to execute the offloading decision.

[0133] Step 4: Update the edge computing task sequence

[0134] If the current task is forwarded to a neighboring satellite, then insert the task back into the edge computing task sequence generated in step 1 according to the time the current task arrives at the neighboring satellite, and proceed to step 2; if the current task is executed, then remove the task from the edge computing task sequence and proceed to step 5.

[0135] Step 5: Calculate the performance metrics of the multi-level unloading strategy

[0136] If an edge computing task is executed, the current network resource utilization is calculated, and the task latency is calculated after the task result is returned. Then, the cost function is calculated based on the above performance indicators to complete the subsequent algorithm performance evaluation.

[0137] Step 5.1 Calculate the latency of edge computing tasks

[0138] Reducing edge computing task latency is a key optimization objective of this invention. The latency considered in this invention mainly includes task upload latency, task queuing latency, task computation latency, and task data return latency. The data transmission rates of the satellite-to-ground communication link and the inter-satellite communication link are respectively determined by v gs and v ss This indicates that the task is located at the edge computing level. i During the upload process, the latency caused by satellite-to-ground data transmission is... The latency caused by inter-satellite data transmission is Among them l gs and l ss These are the average distances for satellite-to-ground and inter-satellite transmission, respectively, and c represents the speed of light. Therefore, Task i The total upload latency can be calculated as follows:

[0139]

[0140] like This indicates that the task has been sent to the cloud computing center for processing. This indicates that the task was executed on a satellite node. 'd' represents the number of inter-satellite transfers involved in the task's unloading process. Task i The formula for calculating the computational delay is:

[0141]

[0142] Where ξ is the CPU computation density, f LEO and f Cloud These represent the computing resources allocated to this task by the satellite and the cloud computing center, respectively. If Task i If the task is processed onboard but onboard computing resources are not yet available, it will be placed in a queue and will not be processed until resources onboard are released, introducing a queuing delay D. i,qThis invention assumes that the cloud computing center has sufficient computing resources, and that if tasks are sent to the cloud computing center for processing, congestion and queuing will not occur. If Task i Unable to reach the time limit L i Once the computation is complete, the computing resources allocated to this task will be released, and the task is considered to have failed to unload; if Task i In L i The data was successfully processed, and the data size was [data missing]. The task results will be returned to the ground user. The formula for calculating the result return latency is as follows:

[0143]

[0144] Based on the above modeling process, the edge computing task can be obtained. i The total processing latency is:

[0145] D i =D i,up +D i,q +D i,C +D i,down

[0146] Step 5.2 Calculate the onboard resource utilization rate

[0147] Because the users served by low-Earth orbit (LEO) satellite constellation networks are unevenly distributed, the computational load on LEO satellites is extremely uneven, leading to inefficient use of onboard computing resources. Therefore, improving the utilization rate of onboard edge computing resources is another important optimization objective of this invention. At time t i For Task i After making the offloading decision, any LEO satellite in the low Earth orbit constellation k It can be represented as:

[0148]

[0149] Among them, CN k It is the current LEO k The number of edge computing tasks being processed, f LEO and These represent the computing resources allocated to a satellite for a single mission and the total onboard computing resources, respectively. Therefore, the overall resource utilization rate of the entire low-Earth orbit satellite constellation network is:

[0150]

[0151] Where numSat represents the total number of satellites in the constellation.

[0152] Step 5.3 Calculate the cost of task unloading

[0153] This invention proposes a multi-level offloading model for edge computing tasks in low-Earth orbit satellite constellation networks. The optimization objective is to reduce task latency and improve the utilization rate of onboard computing resources. Therefore, the cost of the edge computing task offloading strategy can be calculated as follows:

[0154] Cost i =φ1D i +φ2(1-CRUR(t i ))

[0155] φ1 and φ2 are weights used to balance different optimization objectives.

[0156] Step 6: Conduct performance evaluation of the multi-stage offloading strategy for low-Earth orbit satellites.

[0157] This invention evaluates the performance of the multi-level offloading strategy for edge computing tasks in low-Earth orbit satellite constellation networks. The performance metrics considered in this invention include average latency (ATPD) for edge computing tasks, onboard computing resource utilization (CRUR), and average offloading cost (COST) for edge computing tasks. Meanwhile, benchmark algorithms for performance comparison with the strategy proposed in this invention include:

[0158] 1) OTC (Offloading To Cloud): Offload all edge computing tasks to the cloud computing center closest to the user for processing;

[0159] 2) OTAS (Offloading To Access Satellite): Offload all edge computing tasks to the cloud computing center for processing;

[0160] 3) RO (Random Offloading): Randomly select nodes along the transmission path from the access satellite to the cloud computing center to offload the task;

[0161] 4) CDRL (Centralized DRL): A centralized reinforcement learning algorithm that assumes a centralized agent that can acquire global state information in real time and make offloading decisions for edge computing tasks;

[0162] 5) IDRL (Independent DRL): A fully distributed multi-agent reinforcement learning algorithm in which each agent is independent of the others during training and execution.

[0163] This invention tests algorithm performance by varying the number of users in a low-Earth orbit satellite constellation network. Experimental results for three performance metrics—Average Task Latency (ATPD), Onboard Computing Resource Utilization (CRUR), and Edge Computing Task Offloading Cost (COST)—are as follows: Figure 3 , Figure 4 and Figure 5As shown. The MADRL proposed in this invention outperforms the OTC, OTAS, RO, and IDRL algorithms in all performance metrics, and its performance is close to that of the CDRL algorithm. When the user scale is large, its performance is slightly weaker than the CDRL algorithm. However, due to the high dynamic characteristics and large spatial scale of satellite networks, the global network state is difficult to obtain in real time by centralized nodes, thus the CDRL algorithm cannot be implemented and deployed in real systems. This is not a problem for the edge computing offloading strategy based on the MADRL algorithm proposed in this invention, because in actual systems deploying the MADRL algorithm, satellites can make distributed offloading decisions based on local observations, while a centralized architecture is only used during training.

Claims

1. A multi-level offloading method for edge computing in low-Earth orbit satellite constellation networks based on reinforcement learning, the method comprising: Phase 1: Model Training Phase Step 1: Build a user model The ground is divided into M regions with uneven user distribution. Within each region, user locations are uniformly distributed. A probabilistic statistical model is used to characterize user edge computing requests over a period of time. It is assumed that all user behaviors independently and periodically send edge computing task requests to the access satellite. The time interval between two adjacent tasks for a single user follows a negative exponential distribution, with the following probability density function: ； in, The expected time interval for sending requests to a single user. Indicates the time interval between adjacent tasks; Step 2: Generate an edge computing task sequence; Using time series To represent all edge computing tasks, where each element represents a task, and any task... Modeling as ,in, This indicates the size of the data for the task. This is the maximum time limit for uninstalling this task. Points to the user who generated the task. This indicates the time point at which the edge computing task was generated; all tasks are independent of each other, and the data volume of each task follows a truncated normal distribution with an expected value of [value missing]. The variance is ; Step 3: Construct a dynamic topology model of the low-Earth orbit satellite constellation network based on the actual situation; Step 4: Obtain the local observable state of the satellite after receiving the edge computing task based on the topology model obtained in Step 3; Satellites are defined as independent intelligent agents, and their next-hop unloading decisions are made based on locally observable state information. Each low-Earth orbit satellite is treated as an intelligent agent, and its locally observable state space is defined as follows: ;in This refers to the idle state of computing resources on the local satellite and its neighboring satellites (up, down, left, and right). This displays important information about the task currently awaiting uninstallation, including: task data size, maximum task processing time, the number of hops the task has already been uninstalled, and finally... Pointing to the location of the low-orbit satellite; Step 5: The satellite makes an offloading decision based on the local observable state; Each agent receives an edge computing task. And after local information observation, from the action space Choose one action to make an unload decision for the task; the possible actions include local processing. general Forward to neighboring satellites for processing, and... The data is sent to the ground cloud computing center for processing; the current unloading decision neural network is used to calculate the unloading decision. Step 6: Update the edge computing task sequence based on the unloading decision; If the current task is forwarded to a neighboring satellite, the neighboring satellite inserts the task back into the edge computing task sequence generated in step 1 according to the time the current task arrives at the neighboring satellite, and proceeds to step 3; if the current task is executed, the task is removed from the edge computing task sequence and proceeds to step 7. Step 7: Calculate the reward value for the offloading decision made by the satellite; If the edge computing task is executed, the reward value for the multi-level offloading decision will be calculated after the task result is returned to the user. The reward value is then sent to all satellites participating in the unloading task, and training samples are generated and stored in the Replay Buffer. Each agent aims to learn the optimal unloading strategy to reduce unloading costs, i.e., reduce task processing and improve onboard resource utilization. Therefore, before calculating the reward value, the total task latency and the network's edge computing resource utilization need to be obtained. First, the edge computing tasks participating in the collaborative unloading will be... All satellites are modeled as and any one of the satellites The unloading decision model is , Indicates local processing on the satellite , This indicates that the mission will be forwarded to four neighboring satellites. This indicates that the task will be sent to the ground-based cloud computing center; Step 8: Use the decision reward value obtained in Step 7 to adjust the parameters in the unloading decision neural network, and then return to Step 2 until the unloading decision neural network training is complete; Step 9: When performing the actual unloading task, the trained unloading decision neural network is used to calculate the unloading decision.

2. The multi-level offloading method for edge computing of low-Earth orbit satellite constellation networks based on reinforcement learning as described in claim 1, characterized in that, In step 5, the decision neural network used for unloading is the Actor-Critic framework network. The training method in step 8 is as follows: Update the policy network Actor using the deterministic policy gradient DPG algorithm: ； in, This represents the objective function for policy gradient updates. Indicates to Find the gradient. Indicates the expected value of the strategy. Value network right The gradient to be found It is the global status information of the satellite network. It refers to the actions performed by the Actor based on locally observed state variables. The policy network is in a local state of Select action at time The probability, The Critic network represents a centralized value network, and its neural network parameters are as follows: , This is the current policy function; Critic takes global state information and the agent's actions as input, and then outputs the Q-value for each state-action combination; in each round of multi-step update iteration, a target value network is used. and target policy network To fix a target value : ； in, The unloading decision reward value calculated in step 7. It is the action-state combination for the next moment; to alleviate the overestimation problem of the evaluation network, two evaluation networks are used. and To estimate the same set of training samples in parallel, and select the smaller estimate as the final estimate; therefore, the objective of evaluating the network update is modified to: ； Loss function This can be expressed as the following formula: ； With the goal of minimizing the loss function value, gradient descent and backpropagation are used to continuously optimize the parameters of the Actor-Critic framework network in each iteration, enabling the Actor-Critic framework network to more accurately evaluate the agent's policy.

3. The multi-level offloading method for edge computing of low-Earth orbit satellite constellation networks based on reinforcement learning as described in claim 1, characterized in that, The specific method for step 7 is as follows: Step 7.1 Calculate task latency; The data transmission rates of the satellite-to-ground communication link and the inter-satellite communication link are respectively determined by... and Indicates; in edge computing tasks During the upload process, the latency caused by satellite-to-ground data transmission will be... The latency caused by inter-satellite data transmission is ,in and These are the average distances for satellite-to-ground transmission and inter-satellite transmission, respectively. Represents the speed of light; therefore, Total upload latency for: ； like This indicates that the task has been sent to the cloud computing center for processing. This indicates that the task was executed on the satellite node; This indicates the number of inter-satellite transfers required during the unloading process of this mission; computation delay for: ； in, It is the CPU computing density. and These represent the computing resources allocated to this task by the satellite and the cloud computing center, respectively; if If the task is processed onboard but onboard computing resources are not yet available, it will be placed in a queue and will not be processed until resources onboard become available, introducing queuing latency. Assuming the cloud computing center has sufficient computing resources, if tasks are sent to the cloud computing center for processing, no congestion or queuing will occur; if Unable to reach the time limit Once the computation is complete, the computing resources allocated to this task will be released, and the task is considered to have failed to be unloaded; if exist The data was successfully processed, and the data size was [data missing]. The task results will be returned to the ground user, with a delay in return time. for: ； Get edge computing tasks The total processing latency is: ； Step 7.2 Calculate the utilization rate of onboard resources; When low-orbit satellites At any moment for After making the offloading decision, any satellite in the low-Earth orbit constellation Onboard computing resource utilization for: ； in, Is The number of edge computing tasks currently being processed. and These represent the computing resources allocated to a satellite for a single mission and the total onboard computing resources, respectively; therefore, the overall resource utilization rate of the entire low-Earth orbit satellite constellation network is: ； Where numSat represents the total number of satellites in the constellation; Step 7.3 Calculate the unloading decision reward value; intelligent agent is The immediate reward value after making the decision to uninstall the task is ,in, , These are the weights of two different variables in the balanced reward function. express The relative rate of change in resource utilization after the offloading decision is made is calculated using the following formula: ； in, and They are The network resource utilization before and after making the offloading decision; to consider the long-term impact of the agent's decision, the long-term cumulative reward of the action should be further calculated. ,in This is a decay factor that determines the impact of the current unloading decision on future rewards.

Images