Inter-satellite computation offloading and resource allocation method for low-orbit satellites based on reinforcement learning

Abstract

The present invention provides a reinforcement learning-based method for inter-satellite computation offloading and resource allocation for low-orbit satellites. The method involves screening available satellite nodes, selecting edge satellites that meet set conditions and adding them to an edge satellite group; performing optimization modeling, establishing a system model including a task model and a computation model, and using these models to determine optimization problems related to offloading decisions and computational resource allocation. The optimization problem is then modeled using a Markov decision process (MDP) to obtain an MDP model; and finally, using a deep deterministic policy gradient (DDPG) algorithm for joint solution. Finally, the trained DDPG neural network model outputs the optimal offloading strategy and optimal computational resource allocation. This method not only improves computational resource utilization and reduces task execution time, but also reduces overall network energy consumption while ensuring communication quality.

Description

Technical Field

[0001] The present invention relates to a reinforcement learning-based low-orbit satellite inter-satellite computing offloading and resource allocation method, belonging to the technical field of satellite communications. Background Art

[0002] In recent years, with the rapid development of satellite communication technology and the rapid deployment of low-Earth orbit (LEO) satellite networks, LEO satellites have become an important means of providing global internet coverage and enhancing terrestrial communications capabilities. Due to their low orbital altitudes (typically between 300 and 2000 kilometers), LEO satellites offer low communication latency and high data transmission rates. However, LEO satellites also face challenges such as limited resources, constrained computing power, and frequent handoffs, which place higher demands on computational offloading and resource allocation.

[0003] In low-Earth orbit satellite networks, inter-satellite computation offloading (ISCO) involves offloading computational tasks from one satellite to another or to a ground station for processing. This approach fully utilizes the computing resources of each satellite in the constellation, improving overall computing efficiency, reducing latency, and ultimately enhancing service quality. However, efficiently offloading computational tasks and allocating resources in a dynamically changing network environment is a complex and challenging problem.

[0004] Traditional methods usually rely on predefined scheduling strategies and resource allocation algorithms. These methods often perform poorly in the face of dynamically changing and uncertain network environments, and have problems such as low computing resource utilization, long task execution time, and high energy consumption.

[0005] The above problems are issues that should be considered and solved in inter-satellite computing offloading and resource allocation of low-orbit satellites based on reinforcement learning. Summary of the Invention

[0006] The purpose of the present invention is to provide a low-orbit satellite inter-satellite computing offloading and resource allocation method based on reinforcement learning to solve the problems in the prior art of long task execution time, high energy consumption, and the need to improve the utilization of computing resources.

[0007] The technical solution of the present invention is:

[0008] A method for offloading computation and allocating resources between low-orbit satellites based on reinforcement learning, comprising the following steps:

[0009] S1. Screening available satellite nodes: When a local satellite receives a task, it sends information to the cloud center. The cloud center node queries available satellites and receives available satellite information. The available satellite information includes the available computing resources of each available satellite, the number of failures within a pre-set period, and the number of hops between the local satellite and the local satellite. Then, based on the available satellite information, N edge satellites that meet the set conditions are selected and added to the edge satellite group.

[0010] S2. Optimization modeling: Establish a system model including a task model and a computational model. Through the task model and the computational model, obtain the optimization problem related to offloading decision-making and computing resource allocation. Then, perform Markov decision process modeling (MDP modeling) on ​​the optimization problem. The obtained MDP model includes three parts: state, action, and reward.

[0011] S3. Use the DDPG algorithm for joint solution: The obtained task information, task queue status, and edge satellite node information are used as input. During the training phase, the DDPG neural network is trained to obtain a trained DDPG neural network model. During the testing phase, the trained DDPG neural network model is used to output the optimal offloading strategy and optimal computing resource allocation.

[0012] Furthermore, in step S1, N edge satellites that meet the set conditions are selected, specifically, the number of hops required from the local satellite to the edge satellite j ≤ the maximum number of hops from local satellite to edge satellite max , and the average number of failures of edge satellite j within the previous set period σ j ≤The maximum number of recent failures of edge satellitesσ max .

[0013] Furthermore, it is characterized in that: in step S2, a system model is established, specifically,

[0014] S21, in the task model, the total task received is divided into M subtasks, and the task information C is obtained as follows: {C1, ..., C i ,...,C M}, where C i is the information of subtask i, 1≤i≤M, and the information of each subtask i is represented by a tuple consisting of two parameters: C i = <d i ,c i >, where d i represents the amount of data for subtask i, c i Indicates the number of cycles required to complete subtask i;

[0015] S22, the calculation model includes a local calculation model and an offload calculation model,

[0016] The local computation model includes the local computation delay of subtask i when it is executed locally Local computing energy consumption Local trusted value Among them, f L is the local computing power, κ is the energy coefficient, is a constant;

[0017] The offloading calculation model includes the offloading transmission delay of subtask i to edge satellite j for calculation Offload propagation delay Offloading computational latency Offloading transmission energy consumption Offloading computing energy consumption Uninstalling Trusted Values Among them, v ij represents the intersatellite transmission rate, L represents the distance between two adjacent satellites, and c is the speed of light. The computing power allocated to subtask i by the jth edge satellite, p trans is the transmission power between the local satellite and the edge satellite, x is the minimum hop number from the local satellite to the jth edge satellite, and e is the exponential function exp(·); assuming that the probability of node failure follows a Poisson distribution with parameter σ, the unloading credibility value is defined as is the probability value that no failure occurs on the edge satellite node j within the processing time interval [0, t] of subtask i.

[0018] Furthermore, in step S2, the objective function of the optimization problem is established as:

[0019]

[0020] Among them, A is the offloading decision set of all tasks, which is composed of the optimized a ij Composition, F is the computing resource set allocated by the edge satellite to each task, which is composed of the optimized Composition, ω1, ω2, ω3 are respectively represented as the weight factor of delay, the weight factor of energy consumption, and the weight factor of credibility value, and the delay of subtask i Energy consumption of subtask i Credibility value of subtask i

[0021] The constraints are:

[0022]

[0023] a ij (1-a ij )=0

[0024]

[0025] hop j ≤hop max

[0026] σ j ≤σ max ,1≤i≤M,1≤j≤N

[0027] Among them, a ij represents the decision of subtask i on edge satellite j, Indicates that a subtask can only select one edge satellite for offloading, T i Indicates the time the subtask is calculated on the edge satellite. Indicates the maximum acceptable time for processing subtasks. represents the total computing resources allocated by edge satellite j for the subtasks offloaded to this satellite, Indicates the maximum computing power of edge satellite j itself, hop j represents the number of hops between edge satellite j and local satellite, hop max represents the maximum hop limit between the edge satellite and the local satellite, σ j represents the number of failures of edge satellite j in the past time period, σ max Indicates the maximum number of failures of edge satellites in the past time period.

[0028] Furthermore, in step S2, the MDP model is obtained, which includes three parts: state, action and reward. Specifically, the state s p Task information C, task queue status Q T and the remaining computing resources F edge ;Action a p Offload decision and computing resource allocation for the current subtask; the reward is r p =K-(ω1T i +ω2E i +ω3P i )-Q1-Q2, where K is the reward correction factor, T i is the delay of subtask i, E i is the energy consumption of subtask i, P i is the credibility value of subtask i, ω1, ω2, and ω3 are the weight factors of delay, energy consumption, and credibility, respectively. Q1 is the delay penalty factor, and Q2 is the computing resource penalty factor.

[0029] Furthermore, in step S3, in the training phase, the DDPG neural network is trained to obtain a trained DDPG neural network model, specifically,

[0030] S31. Initialize DDPG neural network parameters θ μ ,θ μ' ,θ Q ,θ Q' , learning rate α, β, soft update factor τ, reward discount factor γ, experience replay area R, sampling batch B;

[0031] S32, initialize the environment state value s p =(C,Q T ,F edge ), initialize OU noise;

[0032] S33, based on the task information C and task queue status Q in the current environment T , edge satellite computing resources F edge Outputting Actions from an Actor Target Network

[0033] S34. Execute action a p , calculate the next state s based on the environmental information p+1 and reward r p ;

[0034] S35、(s p ,a p ,r p ,s p+1 ) is stored in the experience playback area R;

[0035] S36. Randomly sample a multidimensional array of batch values ​​from R (s p ,a p ,r p ,s p+1 );

[0036] S37, use the Actor target network to get the action value of the next state;

[0037] S38. Use the Critic target network to obtain the target value of the Q-value function of the current state;

[0038] S39, use Critic training network to get the Q value Q(s) of the current state t ,a t |θ Q );

[0039] S310, updating the Critic training network by minimizing the loss function;

[0040] S311, update the Actor training network through the gradient ascent method;

[0041] S312. Update the Actor target network and the Critic target network through soft update.

[0042] S313, repeat steps S33 to S312 until the task queue Q T If it is empty, the network model will be saved when the total reward score of the current round is greater than the current highest reward value best_score;

[0043] S314, repeat steps S32 to S313 until the maximum number of iterations is reached;

[0044] S315. Output the trained DDPG neural network model.

[0045] Furthermore, in step S3, during the testing phase, the trained DDPG neural network model is used to output the optimal offloading strategy and optimal computing resource allocation, specifically,

[0046] S316, loading the trained DDPG neural network model saved in the training phase;

[0047] S317, initialize the neural network learning rate (α, β), soft update factor τ, reward discount factor γ, and experience replay area R;

[0048] S318, initialize the environment state value s p =(C,Q T ,F edge ), where C is the task information, Q T is the task queue status, F edge Initialize OU noise for edge satellite computing resources;

[0049] S319, based on the task information C and task queue status Q in the current environment T , edge satellite computing resources F edge Outputting Actions from an Actor Target Network

[0050] S320, perform action a p , calculate the next state s based on the environmental information p+1 and reward r p ;

[0051] S321, a p a in ij Join the current uninstall decision A, Add the current computing resource allocation F;

[0052] S322, repeat steps S319 to S321 until the task queue Q TIf it is empty, the current offloading decision A and the current computing resource allocation F are obtained;

[0053] S323. Repeat steps S318 to S322 until the maximum number of iterations is reached, and output the optimal unloading strategy and optimal computing resource allocation for the highest reward action.

[0054] Furthermore, step S34 and step S320 are specifically as follows:

[0055] ①Use a ij 、 Calculate the delay T i Energy consumption E i and the credibility value P i ;

[0056] ② Determine whether the delay constraint is met and computing resource constraints

[0057] ③ If both the delay constraint and the computing resource constraint are satisfied, then r p =K-(ω1T i +ω2E i +ω3P i ), otherwise r p =K-(ω1T i +ω2E i +ω3P i )-Q1-Q2.

[0058] The beneficial effects of the present invention are: this low-orbit satellite inter-satellite computing offloading and resource allocation method based on reinforcement learning, by utilizing the advantages of reinforcement learning, dynamically adjusts the offloading and resource allocation strategies of computing tasks to cope with the complex resource constraints in the low-orbit satellite network. This method can not only improve the utilization rate of computing resources and reduce the execution time of tasks, but also reduce the overall energy consumption of the network while ensuring the quality of communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a flow chart of a method for offloading computation and allocating resources between low-orbit satellites based on reinforcement learning according to an embodiment of the present invention;

[0060] Figure 2 is a schematic diagram illustrating screening of available satellite nodes in an embodiment;

[0061] Figure 3 Schematic diagram illustrating optimization modeling in the embodiment;

[0062] Figure 4 Schematic diagram illustrating the DDPG neural network during the training phase in the embodiment;

[0063] Figure 5 2 is a schematic diagram illustrating the DDPG neural network in the testing phase of the embodiment;

[0064] Figure 6 This is a schematic diagram of reward iteration comparing the embodiment method with three existing methods, including deep Q network DQN, random offloading Random, and local processing Local;

[0065] Figure 7 This is a comparison chart of rewards between the embodiment method and three existing methods, including deep Q network DQN, random offloading Random, and local processing Local, under different task data amounts;

[0066] Figure 8 Schematic diagram of the changes in time efficiency and energy efficiency of the embodiment method under different numbers of edge satellites. DETAILED DESCRIPTION

[0067] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0068] The embodiment provides a method for offloading computation and allocating resources between low-orbit satellites based on reinforcement learning. Figure 1 , including the following steps,

[0069] S1. Screening of available satellite nodes: When a local satellite receives a task, it sends information to the cloud center. The cloud center node sends a query to the available satellites and receives the available satellite information. The available satellite information includes the available computing resources of each available satellite, the number of failures within the pre-set period, and the number of hops between the local satellite and the local satellite. Then, based on the available satellite information, N edge satellites that meet the set conditions are selected and added to the edge satellite group, such as Figure 2 .

[0070] In step S1, select N edge satellites that meet the set conditions, specifically, the number of hops required from the local satellite to the edge satellite j ≤ the maximum number of hops from local satellite to edge satellite max , and the average number of failures of edge satellite j within the previous set period σ j ≤The maximum number of recent failures of edge satellitesσ max .

[0071] S2. Optimization modeling: Establish a system model including a task model and a computational model. Through the task model and the computational model, obtain the optimization problem related to offloading decision and computing resource allocation, and perform Markov decision process modeling (MDP modeling) on ​​the optimization problem. The obtained MDP model includes three parts: state, action, and reward, such as Figure 3 .

[0072] In step S2, a system model is established, specifically,

[0073] S21, in the task model, the total task received is divided into M subtasks, and the task information C is obtained as follows: {C1, ..., C i ,...,C M}, where C i is the information of subtask i, 1≤i≤M, and the information of each subtask i is represented by a tuple consisting of two parameters: C i =<d i ,c i >, where d i represents the amount of data for subtask i, c i Indicates the number of cycles required to complete subtask i.

[0074] S22. Since tasks can be executed locally or at the edge, the computing model includes a local computing model and an offload computing model.

[0075] The local computation model includes the local computation delay of subtask i when it is executed locally Local computing energy consumption Local trusted value in:

[0076] When subtask i is executed locally, the local computing power is f L , from which we can get the local computing delay of the subtask executed on the local satellite

[0077] In order to calculate the energy consumption of local satellites when performing tasks, the energy consumption model E = κf per calculation cycle is adopted. 2 , where κ (energy coefficient) depends on the effective switch capacitance and f is the satellite's operating frequency; therefore, the local computational energy consumption of the local satellite when performing subtask i is

[0078] The trust value in the system is also a factor that affects the selection of nodes. The trust value in local calculation is higher than that in offload calculation, so the trust value in local calculation is set to Its value is set to a constant.

[0079] The offloading calculation model includes the offloading transmission delay of subtask i to edge satellite j for calculation Offload propagation delay Offload calculation delay t ij edge , offloading transmission energy consumption Offloading computing energy consumption Uninstall trusted value P ij edge .in:

[0080] When a subtask cannot meet the conditions for execution on the local satellite, consider offloading the task to the edge satellite for calculation. Offloading subtask i to edge satellite j for calculation will cause offloading transmission delay. Offload propagation delay Right now Among them, v ij represents the inter-satellite transmission rate, x is the minimum number of hops from the local satellite to the jth edge satellite, L is the distance between two adjacent satellites, and c is the speed of light;

[0081] When a subtask is offloaded to the edge satellite side, the edge satellite allocates computing resources to it. Assume that the computing capacity allocated to subtask i by the jth edge satellite is The computational delay of the task unloading at the edge satellite is

[0082] The energy consumption of edge satellites when performing tasks also adopts the energy consumption model of each computing cycle E = κf 2 To calculate, the transmission power between the local satellite and the edge satellite is p trans Therefore, the offload transmission energy consumption of the edge satellite to perform the mission is The energy consumption of unloading calculation is

[0083] Assuming that the probability of a node failure follows a Poisson distribution with parameter σ, the probability of a node failing p times in the time interval [0, t] is Defining Trusted Values The probability that no failure occurs on edge satellite node j within the subtask i processing time interval [0, t] is Where e is the exponential function exp(·).

[0084] In step S2, the optimization problem related to offloading decision and computing resource allocation is obtained, specifically:

[0085] The delay, energy consumption and credibility of subtask i can be obtained from the calculation model formula:

[0086]

[0087] The objective function of the optimization problem is expressed as:

[0088]

[0089] Among them, A is the offloading decision set of all tasks, which is composed of the optimized a ij Composition, F is the computing resource set allocated by the edge satellite to each task, which is composed of the optimized constitute, represents the values ​​corresponding to A and F when the objective function is minimized, ω1 represents the weight factor of delay, ω2 represents the weight factor of energy consumption, and ω3 represents the weight factor of credibility.

[0090] The constraints are:

[0091]

[0092] a ij (1-a ij )=0

[0093]

[0094] hop j ≤hop max

[0095] σ j ≤σ max ,1≤i≤M,1≤j≤N

[0096] Among them: a ij represents the decision of subtask i on edge satellite j, Indicates that a subtask can only select one edge satellite for offloading, T i Indicates the time the subtask is calculated on the edge satellite. It represents the maximum acceptable time for processing subtasks. represents the total computing resources allocated by edge satellite j for the subtasks offloaded to this satellite, Indicates the maximum computing power of edge satellite j itself, hop j represents the number of hops between edge satellite j and local satellite, hop j represents the maximum hop limit between the edge satellite and the local satellite, σ j represents the number of failures of edge satellite j in the past time period, σ max Indicates the maximum number of failures of edge satellites in the past time period.

[0097] In step S2, the MDP model includes three parts: state, action and reward. Specifically, the state s p Task information C, task queue status Q T and the remaining computing resources F edge ;Action a p Offload decision and computing resource allocation for the current subtask; the reward is r p =K-(ω1T i +ω2E i +ω3P i )-Q1-Q2, where K is the reward correction factor, T i is the delay of subtask i, Ei is the energy consumption of subtask i, P i is the credibility value of subtask i, ω1, ω2, and ω3 are the weight factors of delay, energy consumption, and credibility, respectively. Q1 is the delay penalty factor, and Q2 is the computing resource penalty factor.

[0098] In step S2, the optimization modeling consists of three parts: system model, optimization problem and MDP model. In the mission model, the local satellite divides the received mission into M subtasks, each of which is represented by a binary d i ,c i > is used to represent this; since tasks can be executed locally or at the edge, the computing model includes a local computing model and an offloaded computing model. Both models consist of three parts: latency, energy consumption, and trust value. Local computing only involves computational latency and computational energy consumption, while offloaded computing also includes transmission latency, propagation latency, and transmission energy consumption. The optimization problem consists of an objective function and constraints. The independent variables of the objective function include the offloading decision for each task and the computing resources allocated to each task by the edge satellite. The constraints include constraints on the task offloading decision, constraints on the task completion latency, and constraints on the computing power of the edge satellite.

[0099] S3. Use the DDPG algorithm for joint solution: The obtained task information, task queue status, and edge satellite node information are used as input. During the training phase, the DDPG neural network is trained to obtain a trained DDPG neural network model. During the testing phase, the trained DDPG neural network model is used to output the optimal offloading strategy and optimal computing resource allocation.

[0100] In step S3, during the training phase, the DDPG neural network is trained to obtain a trained DDPG neural network model, specifically, as follows: Figure 4 :

[0101] S31. Initialize DDPG neural network parameters θ μ ,θ μ' ,θ Q ,θ Q' , learning rate α, β, soft update factor τ, reward discount factor γ, experience replay area R, sampling batch B;

[0102] S32, initialize the environment state value s p =(C,Q T ,F edge ), initialize OU noise;

[0103] S33, based on the task information C and task queue status Q in the current environment T , edge satellite computing resources F edgeOutputting Actions from an Actor Target Network

[0104] S34. Execute action a p , calculate the next state s based on the environmental information p+1 and reward r p , specifically:

[0105] ①Use a ij 、 Calculate the delay T i Energy consumption E i and the credibility value P i ;

[0106] ② Determine whether the delay constraint is met and computing resource constraints

[0107] ③ If both the delay constraint and the computing resource constraint are satisfied, then r p =K-(ω1T i +ω2E i +ω3P i ), otherwise r p =K-(ω1T i +ω2E i +ω3P i )-Q1-Q2.

[0108] S35、(s p ,a p ,r p ,s p+1 ) is stored in the experience playback area R;

[0109] S36. Randomly sample a multidimensional array of batch values ​​from R (s p ,a p ,r p ,s p+1 );

[0110] S37. Use the Actor target network to get the action value a of the next state t+1 =π(s t+1 |θ μ' ), where a t+1 is the action of the next time step, s t+1 is the next state, π is the strategy, θ μ' Parameters of the Actor target network;

[0111] S38. Use the Critic target network to get the target value of the Q value function of the current state: y t =r t +γQ(s t+1 ,at+1 |θ Q' ), where r t is the reward of the current time step, γ is the reward discount factor, Q(s t+1 ,a t+1 |θ Q' ) represents the Q value of the next state, θ Q' are the parameters of the Critic target network;

[0112] S39, use the Critic training network to directly obtain the Q value Q(s) of the current state t ,a t |θ Q );

[0113] S310, Update the Critic training network by minimizing the loss function: loss function Where N represents the number of edge satellites, y t Represents the target value of the Q-value function of the current state, s t is the current state, a t is the current action, θ Q Parameters for critic training network;

[0114] S311. Update the Actor training network using the gradient ascent method: Where J represents the expected return of the strategy, s t is the current state, a t is the current action, θ Q is the parameter of the Critic training network, θ μ Represents the parameters of the Actor training network, It represents the parameter θ Q The gradient of ▽ a Represents the gradient operation on action a;

[0115] S312. Update the Actor target network and Critic target network through soft update:

[0116] θ μ' =τθ μ +(1-τ)θ μ'

[0117] θ Q' =τθ Q +(1-τ)θ Q'

[0118] Among them, θ μ' is the parameter of the Actor target network, θ μ Represents the parameters of the Actor training network, θ Q'is the parameter of the Critic target network, θ Q is the parameter of the Critic training network, τ is the soft update factor;

[0119] S313, repeat steps S33 to S312 until the task queue Q T If it is empty, the network model will be saved when the total reward score of the current round is greater than the current highest reward value best_score;

[0120] S314, repeat steps S32 to S313 until the maximum number of iterations is reached;

[0121] S315. Output the trained DDPG neural network model including the Acotor target network, the Target_Actor network, the Critic target network, and the Target_Critic network.

[0122] In step S3, during the testing phase, the trained DDPG neural network model is finally used to output the optimal offloading strategy and optimal computing resource allocation, specifically, Figure 5 :

[0123] S316, loading the trained DDPG neural network model saved in the training phase, including the Acotor target network, the Target_Actor network, the Critic target network, and the Target_Critic network;

[0124] S317. Initialize the neural network learning rate (α, β), soft update factor τ, reward discount factor γ, and experience replay area R;

[0125] S318, initialize the environment state value s p =(C,Q T ,F edge ), initialize OU noise;

[0126] S319, based on the task information C and task queue status Q in the current environment T , edge satellite computing resources F edge Outputting Actions from an Actor Target Network

[0127] S320, perform action a p , calculate the next state s based on the environmental information p+1 and reward r p , specifically:

[0128] ①Use a ij 、 Calculate the delay T i Energy consumption E i and the credibility value Pi ;

[0129] ② Determine whether the delay constraint is met and computing resource constraints

[0130] ③ If both the delay constraint and the computing resource constraint are satisfied, then r p =K-(ω1T i +ω2E i +ω3P i ), otherwise r p =K-(ω1T i +ω2E i +ω3P i )-Q1-Q2.

[0131] S321, a p a in ij Join the current uninstall decision A, Add the current computing resource allocation F;

[0132] S322, repeat steps S319 to S321 until the task queue Q T If it is empty, the current offloading decision A and the current computing resource allocation F are obtained;

[0133] S323. Repeat steps S318 to S322 until the maximum number of iterations is reached, and output the optimal unloading strategy and optimal computing resource allocation for the highest reward action.

[0134] In step S3, during the training phase, the agent continuously interacts with the environment and outputs actions based on the current state. The DDPG neural network updates the network parameters through learning experience. After reaching the maximum number of iterations, the trained DDPG neural network model is obtained. In the testing phase, the trained DDPG neural network model finally outputs the optimal offloading strategy and optimal computing resource allocation.

[0135] This reinforcement learning-based method for inter-satellite computation offloading and resource allocation for low-orbit satellites consists of three parts: node selection, optimization modeling, and joint solution. First, during satellite network node screening, after receiving relevant information, the cloud center node selects satellite nodes that meet the requirements and adds them to the edge node group. Secondly, during optimization modeling, the task model and computation model are used to determine the optimization problem related to offloading decisions and computational resource allocation, and this optimization problem is modeled as a Markov decision process. Finally, during the joint solution, task information and edge satellite information are input into the network, and the optimal offloading strategy is determined by continuously interacting with the environment and outputting actions. To address the high cost of single-satellite computation tasks and the varying number of recent failures among satellite network nodes, this method utilizes reinforcement learning to achieve inter-satellite offloading node selection and optimal computational resource allocation while meeting mission time requirements.

[0136] Figure 6 This is a diagram showing the reward iteration comparison between the embodiment method and three existing methods, including the deep Q network DQN, random offloading Random, and local processing Local. Figure 6 It can be seen that as the number of training batches increases, both DDPG and DQN can reach convergence, but the DDPG algorithm in the embodiment can obtain the highest reward among the four algorithms. The reason is that DDPG can obtain accurate resource allocation of edge satellites, improve the utilization of computing resources, and make the latency and energy consumption of computing tasks lower than those of the DQN algorithm.

[0137] Figure 7 This is a comparison chart of rewards between the embodiment method and three existing methods, including deep Q network DQN, random offloading Random, and local processing Local, under different task data amounts. Figure 7 It can be seen from the figure that as the amount of task data increases, the computational delay and energy consumption of the task will increase, resulting in a decrease in the rewards obtained by the four algorithms. However, the task computational delay of the DDPG algorithm in the embodiment is still lower than that of the other three algorithms.

[0138] Figure 8 Schematic diagram of the change of time efficiency and energy efficiency under different numbers of edge satellites in the embodiment method. Figure 8 As can be seen, as the number of edge satellites increases, latency increases while energy efficiency decreases. When the number of edge satellites reaches a certain level, latency decreases but energy consumption increases accordingly. Therefore, good performance is achieved with four edge satellites, and further increases in the number of satellites yield lower performance gains.

[0139] This reinforcement learning-based method for inter-satellite computing offloading and resource allocation for low-orbit satellites first screens available satellites from the satellite constellation through node selection and adds them to the edge satellite group; then establishes a task model and a computing model to solve the optimization problem of minimizing latency and energy consumption and maximizing credibility. Then, by establishing an MDP model and jointly solving it using the DDPG algorithm, it can improve the utilization of computing resources, reduce task execution time, and reduce the overall energy consumption of the network while ensuring communication quality.

[0140] The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A reinforcement learning-based method for offloading computation and allocating resources between low-orbit satellites, characterized by: The following steps are included: S1. Screening available satellite nodes: When a local satellite receives a task, it sends information to the cloud center. The cloud center node queries available satellites and receives available satellite information. The available satellite information includes the available computing resources of each available satellite, the number of failures within a pre-set period, and the number of hops between the local satellite and the local satellite. Then, based on the available satellite information, N edge satellites that meet the set conditions are selected and added to the edge satellite group. S2. Optimization modeling: Establish a system model including a task model and a computational model. Through the task model and the computational model, obtain the optimization problem related to offloading decision-making and computing resource allocation. Then, perform Markov decision process modeling (MDP modeling) on ​​the optimization problem. The obtained MDP model includes three parts: state, action, and reward. In step S2, a system model is established, specifically, S21, in the task model, the total task received is divided into M subtasks, and the task information C is obtained as follows: {C1, ..., C i ,...,C M }, where C i is the information of subtask i, 1≤i≤M, and the information of each subtask i is represented by a tuple consisting of two parameters: C i = <d i ,c i >, where d i represents the amount of data for subtask i, c i Indicates the number of cycles required to complete subtask i; S22, the calculation model includes a local calculation model and an offload calculation model, The local computation model includes the local computation delay of subtask i when it is executed locally Local computing energy consumption Local trusted value Among them, f L is the local computing power, κ is the energy coefficient, is a constant; The offloading calculation model includes the offloading transmission delay of subtask i to edge satellite j for calculation Offload propagation delay Offloading computational latency Offloading transmission energy consumption Offloading computing energy consumption Uninstalling Trusted Values Among them, v ij represents the intersatellite transmission rate, L represents the distance between two adjacent satellites, and c is the speed of light. The computing power allocated to subtask i by the jth edge satellite, p trans is the transmission power between the local satellite and the edge satellite, x is the minimum hop number from the local satellite to the jth edge satellite, and e is the exponential function exp(·); assuming that the probability of node failure follows a Poisson distribution with parameter σ, the unloading credibility value is defined as is the probability value that no failure occurs on the edge satellite node j within the processing time interval [0, t] of subtask i; S3. Use the deep deterministic policy gradient algorithm (DDPG) for joint solution: the obtained task information, task queue status and edge satellite node information are used as input. In the training phase, the DDPG neural network is trained to obtain a trained DDPG neural network model. In the testing phase, the trained DDPG neural network model is finally used to output the optimal offloading strategy and optimal computing resource allocation.

2. The reinforcement learning-based low-orbit satellite inter-satellite computation offloading and resource allocation method according to claim 1, characterized in that: In step S1, select N edge satellites that meet the set conditions, specifically, the number of hops required from the local satellite to the edge satellite j ≤ the maximum number of hops from local satellite to edge satellite max , and the average number of failures of edge satellite j within the previous set period σ j ≤The maximum number of recent failures of edge satellitesσ max .

3. The reinforcement learning-based low-orbit satellite inter-satellite computation offloading and resource allocation method according to claim 1, characterized in that: In step S2, the objective function of the optimization problem is established as: Among them, A is the offloading decision set of all tasks, which is composed of the optimized a ij Composition, F is the computing resource set allocated by the edge satellite to each task, which is composed of the optimized Composition, ω1, ω2, ω3 are respectively represented as the weight factor of delay, the weight factor of energy consumption, and the weight factor of credibility value, and the delay of subtask i Energy consumption of subtask i Credibility value of subtask i The constraints are: a ij (1-a ij )=0 hop j ≤hop max s j ≤σ max ,1≤i≤M,1≤j≤N Among them, a ij represents the decision of subtask i on edge satellite j, Indicates that a subtask can only select one edge satellite for offloading, T i Indicates the time the subtask is calculated on the edge satellite. Indicates the maximum acceptable time for processing subtasks. represents the total computing resources allocated by edge satellite j for the subtasks offloaded to this satellite, Indicates the maximum computing power of edge satellite j itself, hop j represents the number of hops between edge satellite j and local satellite, hop max represents the maximum hop limit between the edge satellite and the local satellite, σ j represents the number of failures of edge satellite j in the past time period, σ max Indicates the maximum number of failures of edge satellites in the past time period.

4. The reinforcement learning-based low-orbit satellite inter-satellite computation offloading and resource allocation method according to claim 3, characterized in that: In step S2, the MDP model includes three parts: state, action and reward. Specifically, the state s p Task information C, task queue status Q T With the remaining computing resources F edge ;Action a p Offload decision and computing resource allocation for the current subtask; the reward is r p =K-(ω1T i +ω2E i +ω3P i )-Q1-Q2, where K is the reward correction factor, T i is the delay of subtask i, E i is the energy consumption of subtask i, P i is the credibility value of subtask i, ω1, ω2, and ω3 are the weight factors of delay, energy consumption, and credibility, respectively. Q1 is the delay penalty factor, and Q2 is the computing resource penalty factor.

5. The reinforcement learning-based low-orbit satellite inter-satellite computation offloading and resource allocation method according to claim 4, characterized in that: In step S3, during the training phase, the neural network used by DDPG is trained to obtain a trained DDPG neural network model, specifically, S31. Initialize DDPG neural network parameters θ μ ,θ μ' ,θ Q ,θ Q' , learning rate α, β, soft update factor τ, reward discount factor γ, experience replay area R, sampling batch B; S32, initialize the environment state value s p =(C,Q T ,F edge ), initialize OU noise; S33, based on the task information C and task queue status Q in the current environment T , edge satellite computing resources F edge Outputting Actions from an Actor Target Network S34. Execute action a p , calculate the next state s based on the environmental information p+1 and reward r p ; S35、(s p ,a p ,r p ,s p+1 ) is stored in the experience playback area R; S36. Randomly sample a multidimensional array of batch values ​​from R (s p ,a p ,r p ,s p+1 ); S37, use the Actor target network to get the action value of the next state; S38. Use the Critic target network to obtain the target value of the Q-value function of the current state; S39, use Critic training network to get the Q value Q(s) of the current state t ,a t |θ Q ); S310, updating the Critic training network by minimizing the loss function; S311, update the Actor training network through the gradient ascent method; S312. Update the Actor target network and the Critic target network through soft update. S313, repeat steps S33 to S312 until the task queue Q T If it is empty, the network model will be saved when the total reward score of the current round is greater than the current highest reward value best_score; S314, repeat steps S32 to S313 until the maximum number of iterations is reached; S315. Output the trained DDPG neural network model.

6. The reinforcement learning-based low-orbit satellite inter-satellite computation offloading and resource allocation method according to claim 5, characterized in that: In step S3, during the testing phase, the trained DDPG neural network model is used to output the optimal offloading strategy and optimal computing resource allocation, specifically, S316, loading the trained DDPG neural network model saved in the training phase; S317, initialize the neural network learning rate (α, β), soft update factor τ, reward discount factor γ, and experience replay area R; S318, initialize the environment state value s p =(C,Q T ,F edge ), where C is the task information, Q T is the task queue status, F edge Initialize OU noise for edge satellite computing resources; S319, based on the task information C and task queue status Q in the current environment T , edge satellite computing resources F edge Outputting Actions from an Actor Target Network S320, perform action a p , calculate the next state s based on the environmental information p+1 and reward r p ; S321, a p a in ij Join the current uninstall decision A, Add the current computing resource allocation F; S322, repeat steps S319 to S321 until the task queue Q T If it is empty, the current offloading decision A and the current computing resource allocation F are obtained; S323. Repeat steps S318 to S322 until the maximum number of iterations is reached, and output the optimal unloading strategy and optimal computing resource allocation for the highest reward action.

7. The reinforcement learning-based low-orbit satellite inter-satellite computation offloading and resource allocation method according to claim 6, characterized in that: Step S34 and step S320 are specifically as follows: ①Use a ij 、 Calculate the delay T i Energy consumption E i and the credibility value P i ; ② Determine whether the delay constraint is met and computing resource constraints ③ If both the delay constraint and the computing resource constraint are satisfied, then r p =K-(ω1T i +ω2E i +ω3P i ), otherwise r p =K-(ω1T i +ω2E i +ω3P i )-Q1-Q2.

Images