Aircraft task offloading method based on DDRL in low earth orbit satellite network

Abstract

The invention relates to a DDRL-based aircraft mission offloading method in a low earth orbit satellite network. The method comprises the following steps: (1) constructing an LEO low earth orbit satellite network system into an LEO satellite mobile edge computing network system; (2) modeling an LEO low earth orbit satellite network system utility maximization problem into a joint decision problem for optimizing task unloading and resource allocation; and (3) converting a joint decision problem into a Markov decision process MDP, learning an optimal task unloading decision by using a dual depth Q network algorithm DDQN under a dual depth reinforcement learning DDRL framework, and obtaining an optimal resource allocation strategy by using a time difference triple strategy gradient algorithm TD3PG on this basis. Simulation experiments show that compared with a reference algorithm, the scheme can effectively access and cooperatively compute concurrent tasks, and has better convergence and superiority under different environment variables.

Description

Technical Field

[0001] The present invention relates to the field of satellite communication technology, and in particular to an aircraft task offloading method based on double deep reinforcement learning (DDRL) in a low-orbit satellite network. Background Art

[0002] Low Earth Orbit (LEO) satellite communications have the advantages of long transmission distance, low latency and cost, and no restrictions on terrain and topography. They can effectively solve the "flight information island" problem of civil aircraft, represented by civil airliners, during navigation. As the amount of business data increases, the central processing and computing latency of traditional satellite networks to transmit data back to the ground for processing is getting longer and longer, which conflicts with the low latency requirements of the business. The emergence of edge computing provides a new model for users to use the computing power of network edge nodes for data processing[1]. In the edge computing model, each satellite is regarded as an edge node to realize on-orbit edge computing. By processing data at the data source, the latency caused by satellite data transmission can be reduced[2].

[0003] In the above context, references [3][4][5][6] studied space-based edge computing from the perspective of architecture and application, and proposed a satellite edge computing architecture, which divides the satellite edge computing system into three parts: user nodes, satellite edge nodes, and ground data centers, which can reduce service delays and save bandwidth. Wang et al. [7] further introduced the corresponding satellite hardware structure and software architecture for transforming traditional satellites into space edge computing nodes, and realized flexible computing services through the virtualization of satellite resources. Xie et al. [8] conducted a comprehensive and detailed analysis of collaborative computing offloading, multi-node task scheduling, and mobility management in the integrated space-ground edge computing network, and pointed out the technical challenges faced by satellite edge computing. Li et al. [9] considered that satellites are edge computing platforms with computing and storage resources, and solved the system service request scheduling decision and service placement problems by solving mixed integer linear programming problems. Cao et al.

[10] proposed a satellite edge cloud architecture based on software-defined networking and network function virtualization, and proposed a computing resource scheduling method that simultaneously considers energy consumption, service delay, and resource utilization. Since the communication, power consumption, and computing resources of satellite platforms are all limited, references

[11]

[12]

[13]

[14] study offloading algorithms that jointly calculate communication resource allocation and offloading decisions, and implement offloading decisions based on deep reinforcement learning, game theory, and heuristic search algorithms, respectively, to reduce system service latency and power consumption.

[0004] However, low-orbit satellite networks still face challenges. Although the deployment of mobile edge computing (MEC) servers on LEO satellites has greatly enhanced their computing capabilities, the computing resources that each satellite can provide are still limited due to the size and energy supply of the satellites. By rationally arranging the computing resources within the satellite constellation, tasks can be transferred to satellites with more idle resources for processing. By efficiently utilizing limited satellite resources, it is not only possible to respond quickly and meet users' requirements for low latency, but also to reduce the workload borne by a single satellite and reduce energy consumption. Therefore, how to reasonably allocate and schedule satellite resources scattered at the edge of the network, realize the direct offloading of aircraft tasks to low-orbit satellites for processing, and reduce the latency and energy consumption during task transmission has become one of the challenges of satellite edge computing. Summary of the invention

[0005] To address the above challenges, the present invention designs a joint optimization scheme of task offloading and resource allocation based on a double deep reinforcement learning (DDRL) framework, aiming to maximize the utility of the LEO satellite network system.

[0006] The method for offloading aircraft tasks based on DDRL in a low-orbit satellite network of the present invention comprises the following steps:

[0007] (1) Build the LEO low-orbit satellite network system into a LEO satellite mobile edge computing network system;

[0008] (ii) Model the utility maximization problem of the LEO satellite network system as a joint decision-making problem of optimizing task offloading and resource allocation;

[0009] (3) The joint decision-making problem is transformed into a Markov decision process (MDP). Under the framework of dual deep reinforcement learning (DDRL), the dual deep Q network algorithm (DDQN) is used to learn the optimal task offloading decision. On this basis, the time-difference triple policy gradient algorithm (TD3PG) is used to obtain the optimal resource allocation strategy.

[0010] The main contributions of the present invention are as follows:

[0011] (1) Aiming at the problem of task offloading and resource allocation in LEO satellite networks, this paper proposes a task offloading and resource allocation model with the goal of maximizing system utility, taking into account factors such as the mobility of low-orbit satellites, differences in satellite computing capabilities, and multi-user tasks.

[0012] (2) Considering that decision offloading and resource allocation is a complex optimization problem with a mixture of discrete variables and continuous variables, the present invention uses the dual deep Q network DDQN algorithm to learn the system environment, obtain the optimal decision offloading in the environment, and then uses the TD3PG algorithm to optimally allocate resources.

[0013] (3) The present invention simulates a LEO satellite network environment on a satellite environment simulation platform to verify the performance of DDRL. The simulation experiment shows that compared with the benchmark algorithm, the proposed scheme can effectively access and coordinate concurrent tasks and has better convergence and superiority under different environmental variables. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It represents the LEO satellite network system;

[0015] Figure 2 Represents the task offloading and resource allocation model;

[0016] Figure 3 Represents the geometric relationship between the satellite and the user;

[0017] Figure 4 Represents the DDQN and TD3PG collaborative training framework;

[0018] Figure 5 Represents the convergence curves of different algorithms;

[0019] Figure 6(a) to Figure 6(c) Respectively represent the performance of different algorithms under different communication resources, where:

[0020] Figure 6(a) shows the reward value, Figure 6(b) shows the system utility, and Figure 6(c) shows the task success rate;

[0021] Figure 7(a) to Figure 7(c) Respectively represent the performance of different algorithms under different computing resources, where:

[0022] Figure 7(a) shows the reward value, Figure 7(b) shows the average energy consumption of the system, Figure 7(c) shows the average delay of the system, and Figure 7(d) shows the task success rate;

[0023] Figure 8(a) to Figure 8(b) Represent the convergence curve and probability density respectively, where:

[0024] Figure 8(a) shows the convergence curves of the three algorithms, and Figure 8(b) shows the system utility probability density. DETAILED DESCRIPTION

[0025] 1. Overview

[0026] Low Earth Orbit (LEO) satellite communications have the advantages of long transmission distance, wide coverage, and no restrictions on terrain and topography, and have become an important means of communication for civil aviation and general aviation. However, the LEO satellite network is a highly heterogeneous and dynamic environment. The mobility of satellite nodes, the complexity of communication links, the uneven spatial and temporal distribution of aircraft, and the coexistence of multiple services have led to many challenging issues in task offloading and resource allocation.

[0027] To this end, the present invention proposes an aircraft task offloading method based on double deep reinforcement learning (DDRL), aiming to maximize the overall utility of the system.

[0028] First, the system utility maximization problem is modeled as a joint optimization problem of task offloading and resource allocation, considering both the computational power and coverage time of the LEO satellites.

[0029] Next, the problem is transformed into a Markov decision process, and the dual deep Q network algorithm DDQN is used to learn the optimal task offloading decision. On this basis, the Temporal Difference Triple Policy Gradient (TD3PG) algorithm is used to obtain the optimal resource allocation strategy.

[0030] Simulation experiments show that under different computing resources and communication resources, the scheme proposed in this invention outperforms other benchmark schemes in terms of system utility, which proves the feasibility of the proposed framework.

[0031] 2. System model and problem modeling

[0032] This section first describes the network model of the LEO satellite network system. Secondly, the communication model and computational model of task offloading and resource allocation based on the association between aviation users and satellites are established. Finally, the optimization problem of minimizing the system energy consumption under constraints is proposed.

[0033] Table 1 lists the key symbols and related descriptions used later.

[0034] Table 1 Important symbols

[0035]

[0036] 2.1 Network Model

[0037] In such Figure 1In the LEO satellite network system architecture shown in Figure 1, the LEO satellite constellation (satellite constellation refers to a group of satellites with a specific working relationship between each other, which work together to complete a specific space mission) is divided into several management domains, each of which is a subnetwork system. The management domain consists of a cluster head satellite and S cluster member satellites equipped with MEC servers. The cluster member satellites are represented by a collection of Indicates that the member satellites There are inter-satellite links (ISL) between the cluster head satellite and member satellites, and between some member satellites. Fixed satellite management nodes face energy loss, spatial position changes, and network replenishment and function updates in the later stage. The current cluster head satellite control and management capabilities fluctuate, and cluster head elections need to be performed periodically to maximize the control capabilities of the cluster head satellite in the system. Therefore, the present invention adopts the multi-factor weighted cluster selection method of "static index + dynamic index" in the document

[15] to periodically elect cluster head satellites. Aircraft users use a collection Taking civil aircraft as an example, aircraft users The number of aircraft users is represented by U. Assuming that the aircraft is located in a remote area, the aircraft can only obtain services by accessing the LEO satellite mobile edge computing network

[16] . When the aircraft enters a LEO satellite constellation management domain, it will automatically access the cluster head satellite of the management domain. The cluster head satellite will unload the task to the member satellites for processing according to the type of task. It should be noted that if the aircraft is covered by multiple management domains, the cluster head satellite of the management domain with high channel quality will be selected by default for access.

[0038] The LEO satellite network system operates in time slot mode, using a collection Indicates that the time gap Time is divided into Y time slots of equal length, each time slot is τ. Assume that the routing table and trajectory information of the LEO constellation are known to the cluster head satellite. The link status between satellites usually depends on the relative position between the satellite and the earth

[17] , and when the satellite is located in the non-polar region of the earth, the link status of the ISL is relatively stable. Based on this, in order to simplify the model, we assume that the LEO satellite constellation has a stable topological structure and connectivity within a certain period of time

[18] . At the beginning of each time slot, the aircraft will randomly and parallelly generate tasks, which are independent and indivisible. Assume that the time interval between task arrivals follows an exponential distribution with a mean value of μ. In time slot t, the task generated by user u in time slot t is represented by the triple Indicates that, among them, l u,t and c u,t represents the data size and workload of the task, is the maximum tolerable delay of the task. In addition, the total number of tasks uploaded by users during the operation of the system is represented by K. The output data of the task is usually small compared to the input data

[19] , so the delay in returning the calculation result can be ignored. In addition, assuming that when the cluster head satellite offloads the task to the member satellite, the user identity, communication protocol and other information are usually sent together as part of the task assignment message. After completing the processing of several time slots, the member satellite can configure according to this information, establish a communication link with the user, and then return the processing result.

[0039] like Figure 2 As shown in Figure 1, the model of task offloading and resource allocation in the LEO satellite network system includes three steps: aircraft access and transmit tasks to the cluster head satellite for queuing, cluster head satellite unloads tasks from the waiting queue to member satellites, and tasks are processed on member satellites. First, when the cluster head satellite receives a large number of user tasks, it is necessary to consider the waiting queue when distributing and unloading tasks. The cluster head satellite puts the tasks to be unloaded into the unloading queue N(t). Secondly, let the decision matrix symbol of the cluster head satellite to unload tasks to member satellites be Represents. Matrix Indicates whether to offload the task of the i-th user to the j-th satellite for processing. Indicates that the task of the i-th user is offloaded to the j-th satellite for processing. Indicates that the task of the i-th user is not offloaded to the j-th satellite for processing. Finally, when the task is offloaded to the member satellite s by the cluster head satellite, s needs to be Allocate computing resources. Set the task The computing resource allocation strategy is used It indicates that The computing resources provided are And there is represents the maximum computing resources owned by a member satellite s. In addition, the present invention adopts a reservation-based resource allocation mechanism

[20] . After a task is unloaded to a satellite, the orthogonal computing resources allocated for the task will be reserved and occupied for several time periods until the task calculation is completed. This method ensures that the task is processed immediately when it arrives at the satellite without additional delay jitter

[21] . Therefore, due to the different task processing processes on different satellites, the available computing resources of the satellite for the current task are also different. The computing resources allocated by the satellite in time slot t cannot exceed the current available resources, which can be expressed as

[0040]

[0041] Among them, φ s,t Indicates the resources that have been occupied by satellite s in time slot t.

[0042] 2.2 Communication model

[0043] (1) Satellite-aircraft communication model

[0044] The user uploads the task through the link between the cluster head satellite and the transmission link uses the Ka band. Here, the present invention considers the quasi-static fading channel model in a time slot according to the literature

[22] . Specifically, the signal y received by the cluster head satellite from user u in time slot t is u,t Expressed as

[0045]

[0046] Among them, p u and u Represent the transmission power and data signal of user u respectively. 2 ) is the additive white Gaussian noise (AWGN) of the channel, σ 2 is the noise variance. h u,t represents the AWGN channel between user u and the cluster head satellite in time slot t, which is expressed as

[0047] h u,t =δ u η u (d u,t ) -β (3)

[0048] Among them, δ u ~(0,1) is a complex Gaussian variable with Rayleigh fading, and η u is the shadow fading that follows a log-normal distribution. u,t is the actual distance between user u and the cluster head satellite in time slot t, which is obtained from the geometric relationship between user u and the cluster head satellite, as follows: Figure 3 As shown. Since there is a difference of two or more orders of magnitude between the height of low-orbit satellites and the height of general aircraft, this paper does not consider the height of the aircraft from the ground, and similarly does not consider the speed of the aircraft. Assume that the user can only start transmitting requests at a certain elevation angle α, where α is the minimum angle symmetrical with respect to the line connecting the center of the earth and the user's position. Let θ represent the central angle corresponding to the arc covered by the satellite, R is the radius of the earth, and H is the satellite orbit altitude. Ignoring other factors, assuming that α is known, the relationship between θ and α can be expressed as

[0049]

[0050] Therefore, the actual distance between the cluster head satellite and user u at time slot t is

[0051]

[0052] In addition, β is the path loss exponent, which means that the communication condition between the satellite and the user is negatively correlated with the distance between the two.

[0053] To simplify the problem, it is assumed that the satellite spectrum is orthogonally allocated to all users, and the total communication bandwidth of the cluster head satellite is evenly distributed among the currently associated users. Therefore, according to the Shannon formula, the data transmission rate from user u to the cluster head satellite can be expressed as:

[0054]

[0055] Where B represents the total bandwidth in the Ka band. Obviously, the worse channel h u,t A larger number of associated users U will result in a slower data transmission rate.

[0056] Considering the delay problem, there will be communication delay inside the machine when the data packet is pushed to the link, and there will be propagation delay outside the machine when the electronic signal is transmitted. Let c represent the speed of light, then the time delay between user u and the cluster head satellite at time slot t is

[0057]

[0058] In addition, the task

[0059] The transmission energy consumption from user u to the cluster head satellite is

[0060]

[0061] (2) Inter-satellite communication model

[0062] N(t) is used to store the tasks that have been uploaded to the cluster head satellite in time slot t but have not been unloaded. The unloading queue N(t+1) in the next time slot is obtained by subtracting the amount of unloaded task data B(t) and adding the amount of task data newly uploaded to the cluster head satellite. get:

[0063]

[0064] in, Represents the relationship between time t and task The relevant task arrival rate follows the Poisson distribution, and τ is the length of a time slot. Assuming that the CPU frequency of the cluster head satellite is f′ when processing the unloading queue in time slot t, the amount of task data processed by the cluster head satellite in the unloading queue in one time slot is

[0065]

[0066] Where J represents the number of CPU cycles required to process each bit of task data. Therefore, the queue delay of the unloading queue N(t) can be expressed as

[23]

[0067]

[0068] Among them, X N(t) Indicates the queue length, It represents the average task arrival rate of queue N(t), which can be expressed as

[0069]

[0070] In addition, energy consumption is not considered during task queuing.

[0071] To improve real-time performance, it is assumed that the task can only be unloaded once from the cluster head satellite to the member satellite s, and there is no multi-hop unloading process between member satellites. It will be unloaded from the unloading queue of the cluster head satellite to the member satellite for processing. Therefore, the time delay of user u's task from the cluster head satellite to the unloading satellite in time slot t is expressed as

[0072]

[0073] in, represents the dynamic distance between the member satellite s and the cluster head satellite in time slot t. At the same time, let p be the transmission power of the cluster head satellite, then the task The transmission energy consumption can be expressed as

[0074]

[0075] 2.3 Computational Model

[0076] In time slot t, the computing resources allocated to user u by unloading satellite s are It can be seen that the processing task The calculation time is

[0077]

[0078] In this stage, the task calculation delay is mainly considered, and the queuing delay before the current task calculation is negligible.

[0079] The present invention uses a widely adopted energy consumption model based on a computing cycle. The processing energy consumption of a task on the satellite is calculated as e = κf 2

[24] , where κ is the energy consumption coefficient, which depends on the effective switch capacitance of the chip architecture, and f is the CPU frequency. Then, to execute a task on the unloaded satellite s The computational energy consumption is expressed as

[0080]

[0081] 2.4 Problem Modeling

[0082] Based on the above discussion, the task The total delay from access to unloading is T u,t It is composed of (7), (11), (13) and (15) and is expressed as

[0083]

[0084] And there is That is, the end-to-end delay of each task cannot exceed the maximum tolerable delay. This condition is used as a sign of whether the task is completed or not, and a binary variable a is used to indicate whether the task is completed or not. u,t It means that

[0085]

[0086] If the delay constraint is met, the task is considered to be successfully processed. u,t =1, otherwise the task is considered failed. u,t =0.

[0087] The system energy consumption generated in this process is composed of (8), (14) and (16), which can be expressed as

[0088]

[0089] Therefore, the task The end-to-end delay and system energy consumption are determined by the joint strategy of task offloading and resource allocation. At the beginning of each time slot, the cluster head satellite decides to which member satellite to offload the task and how much computing resources to allocate for the task.

[0090] The system cost includes the sum of the delay and energy consumption of all tasks during the system operation. The joint optimization of task offloading and resource allocation can be described as the system cost minimization problem, which is expressed as

[0091]

[0092] Among them, ξ∈[0,1] is used to weigh the proportion of energy consumption cost and delay cost in the system cost.

[0093] The minimization of system cost is replaced by the maximization of system utility, which is modeled as

[0094]

[0095] Among them, λ is an overhead factor, which is used to solve the problem that the task delay and energy consumption cannot be taken into account due to unfinished tasks. λ is a constant and sufficiently large. Constraints (21a) and (21b) indicate that each user can offload at most one task to one LEO satellite; (21c) indicates that the resources allocated by the satellite to the task cannot exceed the maximum resources owned by the satellite; (21d) indicates that the sum of the computing resources allocated by each satellite to the current offloaded task cannot exceed the current available resources of the satellite.

[0096] 3. Algorithm Design

[0097] According to the above discussion, is a discrete variable, It is a continuous variable. The problem to be solved by the present invention is a complex optimization problem of a mixture of discrete variables and continuous variables. Therefore, in this section, a DDRL framework is proposed to solve the joint decision-making problem. For the discrete variable environment, DDQN is used to learn the system environment to obtain the optimal decision unloading in the environment. Then, for the continuous variable environment, TD3PG is used to allocate resources to obtain the optimal resource allocation strategy. The specific process is described as follows.

[0098] 3.1 Problem Transformation

[0099] Considering that decision offloading is related to resource allocation problems, that is, resource allocation and task offloading are coupled, this paper deals with these two problems jointly.

[0100] Due to the correlation of decisions in adjacent time slots, the task offloading and resource allocation process in the above dynamic scenario can be transformed into an MDP problem. MDP consists of five elements: state space S, action space A, state transition probability P, reward function R, and discount factor γ. At time slot t, the cluster head satellite can obtain the load status, remaining resources and other status information of the member satellites and the uploaded user task information. Then the cluster head satellite outputs the task offloading and resource allocation decision, and returns the resource allocation decision to the member satellite while making the offloading action. The member satellite allocates resources for task processing. Finally, the member satellite returns the result to the user. The cluster head satellite learns and implements strategies through continuous interaction with the system environment, and obtains rewards to improve the strategy. The description of the key elements in MDP is as follows:

[0101] State space: At time slot t, the cluster head satellite has the state information of all aircraft users and cluster member satellites from a holistic perspective. The system state is expressed as L u,t represents the geographical location of user u at time slot t. The location information is described as a two-dimensional vector, including the longitude and latitude in the geodetic coordinate system, and the elevation is set to 0. u,t is a U-dimensional vector, each element of which is the distance between user u and the cluster head satellite at time slot t. Represents the attribute information of the task uploaded by user u in time slot t. is an S-dimensional vector, which refers to the distance between satellite s and the cluster head satellite at time slot t. s,t It is also an S-dimensional vector, indicating the number of resources currently occupied by each member satellite. N(t) represents the queue status of the current unloading queue.

[0102] Action space: The action refers to the selection of unloading satellites and resource allocation scheme made by the cluster head satellite in time slot t, which is defined as in and They represent task offloading actions and resource allocation actions respectively. In the process of dynamic task offloading and resource allocation, the cluster head satellite needs to observe the system environment S t Determine the satellite to be unloaded and make an unloading decision based on the system environment observation information S t , determine the unloading of satellite as the task Allocated computing resources.

[0103] Reward: In system state S t Perform tasks for users

[0104] Take action A u,t After that, the system environment will enter a new state S t+1 And return the corresponding reward. Put the system in state S t The total cost under t ) indicates that when the system undergoes action A u,t Enter state S t+1 After that, state S t and S t+1 There is a cost difference ψ(S t )-ψ(S t+1 ). Therefore, the design of the reward function is based on the cost difference of the system state transition process, and the specific formula is expressed as

[0105] r(S t ,A t ,S t+1 )=ψ(S t )-ψ(S t+1 ) (twenty two)

[0106] When r(S t ,A t ,S t+1 )>0, that is, when the system cost decreases with time slots, the action will get a positive reward value, otherwise the reward value is negative, indicating that the action is punished.

[0107] 3.2 Model Training

[0108] The proposed DDRL framework consists of two modules: DDQN and TD3PG. DDQN is used for offloading decision problems, and TD3PG is used for computing resource allocation problems. The detailed interaction process between user tasks and the environment is shown in Figure 4 As shown, there are five steps:

[0109] Step ①: DDQN takes the environment state as input and iteratively provides services to each user;

[0110] Step ②: The DDQN neural network processes the information and outputs appropriate offloading decisions for each user;

[0111] Step ③: The environment state and the user’s uninstallation decision are processed as input by the TD3PG algorithm;

[0112] Step ④: TD3PG considers the above information and then gives appropriate computing resource allocation;

[0113] Step ⑤: DDRL outputs the final joint decision.

[0114] Specifically, each interaction first performs a task offloading action through DDQN Then perform resource allocation actions according to TD3PG In addition, the task From state S t Action A u,t Enter state S t+1 The reward value r(S t ,A t ,S t+1 ) with the symbol r u,t When DDQN interacts with the environment, it follows the ε-greedy strategy to execute the action of the offloading decision. ε is a probabilistic decision factor ε, which is between 0 and 1. During the DDQN learning process, ε is used to represent the probability of randomly selecting an action, and 1-ε is used to represent the execution of the action given by the DDQN main network. probability.

[0115] When exploring the environment for the first time, a larger value is set for ε, which can bring more randomness and help discover new states and actions. As learning accumulates, the value of ε will gradually decrease, and the agent will make more use of the learned knowledge and choose the action that seems to have the greatest reward. On the other hand, since TD3PG is suitable for continuous actions Therefore, its exploration method adds random noise. This noise follows a normal distribution with variance σ, and the value of σ gradually decreases during the learning process.

[0116] After completing the interaction, Algorithm 1 generates an observation for DDQN and an observation for TD3PG. These observations are stored in two independent replay buffers B1 and B2, respectively.

[0117] Algorithm 1 Sample generation for task and environment interaction

[0118]

[0119] 3.3DDRL Joint Optimization Framework

[0120] Task offloading and resource allocation based on DDRL is described in Algorithm 2, and its sample data is collected from Algorithm 1. The beginning of Algorithm 2 is the update process of the DDQN algorithm. This part sets up the Q network and the target network DDQN collects quadruplets from replay buffer B1 For Q and When the playback buffer B1 is full, the latest observations will replace the old observations. When using samples to update Q, DDQN Calculate the value y u,t As the target value, it is expressed as

[0121]

[0122] Where γ is the decay factor. The loss function for updating Q is the square error function, defined as

[0123]

[0124] Next, use the above formula to calculate the gradient and update the Q network every g1 steps. At the same time, by setting Update every G1 step

[0125] The second part of Algorithm 2 is the update process of the TD3PG algorithm. After making an offloading decision for a task, the algorithm allocates the optimal amount of computing resources to the task. TD3PG is an Actor-Critic method that contains two independent value networks and a policy network and its corresponding target network. Actor network A predicts resource allocation strategy This is a continuous value of a computational resource. Critc networks C1 and C2 determine the q-value of the strategy generated by the Actor network during the learning process. Before TD3PG training, the algorithm samples some values ​​from the playback buffer B2. During training, the algorithm samples some values ​​from the target network. and Calculate the target value

[0126]

[0127] in Then, the target value y u,t Compared with the q value, the difference between the two is the time difference (TD) error δ. This function is a square error, defined as

[0128]

[0129] The gradient of the above equation will be used to update the Critic networks C1 and C2. To update the Actor network A, the algorithm calculates the gradient of the value q1=C1(s,a) predicted from the Critic network C1. Since the Actor network A remains fixed when updating the Critic network C1, the algorithm uses the gradient ascent method (minimize -q1) to maximize the q1 value. The training networks A, C1, and C2 are updated every g2 steps. At the same time, the target network and They are updated every G2 steps by setting them to be the same as the training network. In addition, to ensure the stability of the policy network update, the policy network is updated every μ times the value network is updated.

[0130] Algorithm 2: Task offloading and resource allocation algorithm based on DDRL

[0131]

[0132]

[0133] 3.4 Time Complexity Analysis

[0134] First, the total number of training rounds determines the number of times the algorithm needs to iterate. Each iteration means a complete training process, including obtaining feedback from the environment and updating the strategy. Obviously, more training rounds will directly increase the overall computational complexity of the algorithm. We use |episode| to represent the size of the total number of training rounds, and the time complexity is linear with |episode|. Secondly, the environment termination time Y represents the upper limit of the number of steps the environment runs in each training round. Each step involves processes such as state transfer, action selection, and reward calculation. With each additional step, the algorithm's computational complexity will increase accordingly, so the time complexity is also linear with Y. Finally, the number of user tasks K is described. K represents the number of tasks that need to be processed in each training step. In each step, the algorithm needs to calculate and make decisions for each user task. Therefore, the increase in K will linearly increase the computational complexity of each step, which ultimately affects the overall time complexity. In summary, the time complexity of DDRL is O(|episode|YK).

[0135] 4. Experimental Design and Results Analysis

[0136] This section uses simulation methods to verify the effectiveness of the proposed method. The method in references

[25]

[26]

[27] is simulated and verified based on the AGI STK (System Tool Kit) software platform. First, a satellite cluster based on the Iridium constellation is set up, and then the users are randomly placed under the coverage of the satellite cluster. Finally, the satellite orbit parameters are set to be closer to the actual scenario. We set the time span of the STK scenario to one hour. The satellite trajectory data samples are obtained at a time interval of τ=1.

[0137] In the proposed DDRL-based framework, the DDQN training network Q contains a hidden layer with 256 neurons. The TD3PG training networks A, C1 and C2 also have a hidden layer with 256 neurons each. and They have the same structure as their corresponding training networks. In Algorithm 2, we set the number of sampling rounds g1=g2=4, the number of update rounds G1=1000, G2=100. In addition, the playback buffers B1=12000, B2=60000, and the total number of rounds episode=4000. After the system environment is started, DDQN first learns, and TD3PG will start learning when its buffer collects 12000 samples, because TD3PG needs more samples to stabilize convergence. DDRL is optimized using the Adam optimizer with an initial learning rate of 0.00015. After 1000 rounds, the learning rate drops by 90%. In addition, the decay factors of DDQN and TD3PG are set to 0.9 and 0.99, respectively. The ε value is initially set to 1.0 and decreases linearly to 0.02. TD3PG uses a normal distribution to explore, and the variance of the normal distribution is multiplied by 0.99 after each update.

[0138] In addition, the total bandwidth B in the simulation environment is set to 300 MHz, and the transmission power of the user and the satellite is p u and The maximum computing resource satellite owned by member satellite s is 2W and 5W respectively

[28] . The frequency band is 2 to 4 GHz [5]. The distance between satellites and the communication capacity are 800 to 1200 km and 10 Gbps respectively. In addition, the weight parameter ξ in the objective function is set to 0.6. Table 2 summarizes the main parameter settings in the simulation.

[0139] Table 2 Simulation parameters

[0140]

[0141] In order to objectively evaluate the performance, we selected three representative task offloading and resource management strategies for comparison, including:

[0142] Baseline method-1: DQN-DDPG algorithm. Considering that the scenario set in this paper is a complex environment that contains both discrete and continuous action spaces, and the DQN and DDPG algorithms are classic DRL algorithms that act on discrete action spaces and continuous action spaces respectively, we combined the two algorithms and compared them with our solution.

[0143] Baseline method-2: Heuristic algorithm based on greedy offloading strategy

[29] . This strategy follows the principle of greedy algorithm and only selects the decision that seems to be the best at each step without considering the overall optimal solution.

[0144] Baseline method-3: Random method. Tasks are randomly offloaded to a member satellite and resources are randomly allocated without considering whether the delay and resource constraints are met.

[0145] 4.1 Convergence Analysis

[0146] In this section, the convergence performance of the four algorithms is evaluated by calculating the average cumulative reward. Figure 5 It can be observed that both the proposed method and the baseline method-1 achieved stable task offloading and resource allocation processes in the final training process. Compared with the baseline algorithm-1, the proposed method obtained a better cumulative reward value. This is because the DDQN and TD3PG algorithms in the proposed method optimize the over-estimation problems existing in the DQN and DDPG algorithms respectively, which can reduce the oscillation and instability in the learning process. In addition, the baseline algorithm-2 and the baseline algorithm-3 are not affected by the training process and have poor performance, but the baseline algorithm-2 can also find the optimal solution under certain circumstances, and its performance is better than that of the baseline algorithm-3.

[0147] 4.2 Impact of bandwidth and computing resources on performance

[0148] first, Figure 6(a) to Figure 6(c)The performance of each strategy is shown when the satellites have different total bandwidths B. It can be observed from Figure 6(a) and Figure 6(c) that when the total bandwidth B is small, the reward rate and mission success rate of each algorithm are very low. This is because all aircraft users uniformly access the cluster head satellite and occupy the bandwidth evenly, resulting in data transmission at a smaller communication rate, increasing transmission delay and energy consumption. As B gradually increases, mission data can be transmitted through channels with richer communication resources, which can ensure that more tasks are completed on time and increase system utility, as shown in Figure 6(b) and Figure 6(c). When B increases further, most tasks can be successfully completed, but the impact on system utility gradually decreases, and the rewards of each algorithm begin to stabilize. In addition, compared with the three baseline algorithms, the performance of the proposed method is the best, which means that the task offloading and resource allocation strategies based on DDRL can find the best offloading satellite and the optimal resource allocation strategy within the dynamic communication bandwidth range.

[0149] In addition, this experiment compares satellites with different computing capabilities When the joint strategy changes, such as Figure 7(a) to Figure 7(d) As shown in Figure 7(a), when When is small (resources are scarce), tasks often cannot be assigned appropriate resources for processing and cannot be processed within the specified time, resulting in a very large penalty in the learning process. As , more tasks can be successfully processed, and the reward will also increase positively. When it increases further within a certain range, the system The demand has been basically met, so the success rate of the mission will not be significantly improved at this time. On the contrary, the continuous increase of The value will lead to greater computing energy consumption and exponential growth, which leads to a decrease in the reward value in the second half. As can be seen from Figure 7(b) and Figure 7(c), the increase in computing resources will also lead to a rapid increase in energy consumption, resulting in an increase in system energy consumption, but the system latency is reduced. At the same time, the method proposed in this paper still maintains the best performance. In addition, as can be seen from Figure 7(d), the success rate of each algorithm task increases with The increase in computing power can reduce processing latency, thereby improving the success rate of tasks. Among them, the DDRL method proposed in this paper maintains the maximum reward, while the performance of the benchmark algorithm-1, benchmark algorithm-2 and benchmark algorithm-3 decreases in turn.

[0150] 4.3 Ablation Experiment

[0151] In order to verify the effectiveness of DDRL, two modified joint optimization algorithms are established to conduct ablation experiments:

[0152] (1) DDQN-DDPG algorithm: The DDPG algorithm is used instead of the TD3PG algorithm in resource allocation decisions. The DDPG algorithm does not use the dual critic architecture and does not adopt the method of delayed policy updates.

[0153] (2) DQN-TD3PG algorithm: The DQN algorithm is used instead of the DDQN algorithm in task offloading decision making. Specifically, we use the same network for action selection and evaluation, instead of using a separate target network for action value evaluation.

[0154] As shown in Figure 8(a), the average cumulative rewards of DDQN-DDPG and DQN-TD3PG after convergence are 5000 and 4000 respectively, both lower than 7000 of DDQN-TD3PG. This is because DDQN and TD3PG usually have higher performance and faster convergence speed than DQN and DDPG when dealing with tasks that require highly accurate value estimation. In addition, the average cumulative reward of DDQN-DDPG is slightly higher than that of DQN-TD3PG, because the input of the continuous action space algorithm requires the output of the discrete action space algorithm as a prerequisite, and DDQN has better performance and higher robustness than DQN.

[0155] Figure 8(b) shows the probability density of the system utility distribution of the three algorithms, where the system utilities of DQN-TD3PG, DDQN-DDPG, and DDQN-TD3PG are concentrated around 3600, 3800, and 4100, respectively, and the curve width of DDQN-TD3PG is narrower. The mathematical expectations and variances corresponding to the three curves show that DDQN-TD3PG has the best joint optimization performance.

[0156] 5. Conclusion

[0157] The present invention proposes a DDRL framework based on a dual deep learning algorithm for task offloading and resource allocation of edge computing of low-orbit satellites. The framework contains two DRL algorithms to solve the problem of limited communication and computing resources of aircraft users during task offloading and resource allocation. The effectiveness of the proposed scheme is proved by analyzing the convergence of the proposed scheme and its performance under different resources. Finally, the ablation experiment is used to prove that the proposed scheme has a better system utility than other DRL combinations. In the next stage, it can be studied to divide a task into a certain number of small tasks, and then offload them to different satellites for distributed processing.

[0158] References

[0159] [1]Mao Y,You C,Zhang J,et al.A survey on mobile edge computing:Thecommunication perspective[J].IEEE communications surveys&tutorials,2017,19(4):2322-2358.

[0160] [2]Denby B,Lucia B.Orbitaledge computing:Machine inference in space[J].IEEE Computer Architecture Letters,2019,18(1):59-62.

[0161] [3]Wei J,Han J,Cao S.Satellite IoT edge intelligent computing:Aresearch on architecture[J].Electronics,2019,8(11):1247.

[0162] [4]Cao S,Zhao Y,Wei J,et al.Space-based cloud-fog computingarchitecture and its applications[C] / / 2019IEEE World Congress on Services(SERVICES).IEEE,2019,2642:166-171.

[0163] [5]Guo J,Du Y.Fog service in space information network:Architecture,use case,security and challenges[J].IEEE Access,2020,8:11104-11115.

[0164] [6]Wei J,Cao S.Application of edge intelligent computing in satelliteInternet of Things[C] / / 2019IEEE international conference on smart internet ofthings(SmartIoT).IEEE,2019:85-91.

[0165] [7]Wang Y,Yang J,Guo X,et al.Satellite edge computing for theinternet of things in aerospace[J].Sensors,2019,19(20):4375.

[0166] [8]Xie R,Tang Q,Wang Q,et al.Satellite-terrestrial integrated edgecomputing networks:Architecture,challenges,and open issues[J].Ieee Network,2020,34(3):224-231.

[0167] [9]Li C,Zhang Y,Hao X,et al.Jointly optimized request dispatching andservice placement for MEC in LEO network[J].China Communications,2020,17(8):199-208.

[0168]

[10] Cao B,Zhang J,Liu X,et al.Edge–cloud resource scheduling inspace–air–ground-integrated networks for internet of vehicles[J].IEEEInternet of Things Journal,2021,9(8):5765-5772.

[11] Cui G,Long Y,Xu L,etal.Joint offloading and resource allocation for satellite assisted vehicle-to-vehicle communication[J].IEEE Systems Journal,2020,15(3):3958-3969.

[0169]

[12] Wang B,Feng T,Huang D.A joint computation offloading and resourceallocation strategy for LEO satellite edge computing system[C] / / 2020IEEE 20thInternational Conference on Communication Technology(ICCT).IEEE,2020:649-655.

[0170]

[13] Cui G,Li X,Xu L,et al.Latency and energy optimization for MECenhanced SAT-IoT networks[J].IEEE Access,2020,8:55915-55926.

[0171]

[14] Tran T X,Pompili D.Joint task offloading and resource allocationfor multi-server mobile-edge computing networks[J].IEEE Transactions onVehicular Technology,2018,68(1):856-868.

[0172]

[15] Li Ning, Yue Chengfei, Guo Haibo, et al. Design of domain control strategy for large-scale satellite constellations[J]. Chinese Space Science and Technology, 2023, 43(01): 18-28.

[0173]

[16] Casoni M, Grazia CA, Klapez M, et al. Integration of satellite and LTE for disaster recovery [J]. IEEE Communications Magazine, 2015, 53(3): 47-53.

[0174]

[17] Wang F, Jiang D, Qi S, et al. Fine-grained resource management for edge computing satellite networks [C] / / 2019 IEEE Global Communications Conference (GLOBECOM). IEEE, 2019: 1-6.

[0175]

[18] Su Y, Liu Y, Zhou Y, et al.Broadband LEO satellite communications: Architectures and key technologies[J]. IEEE Wireless Communications, 2019, 26(2):55-61.

[0176]

[19] Tang Q, Fei Z, Li B, et al. Computation offloading in LEO satellite networks with hybrid cloud and edge computing[J]. IEEE Internet of ThingsJournal, 2021, 8(11):9169176.

[0177]

[20] Cui G, Long Y, Xu L, et al.Joint offloading and resource allocation for satellite assisted vehicle-to-vehicle communication[J]. IEEE SystemsJournal, 2020,15(3):3958-3969.

[0178]

[21] De Sanctis M,Cianca E,Araniti G,et al.Satellite communicationssupporting internet of remote things[J].IEEE Internet of Things Journal,2015,3(1):113-123.

[0179]

[22] Cao X,Yang B,Shen Y,et al.Edge-assisted multi-layer offloadingoptimization of LEO satellite-terrestrial integrated networks[J].IEEE Journalon Selected Areas in Communications,2022,41(2):381-398.

[0180]

[23] Soni G,Sharma M.Performance Evaluation of a Free Space OpticalLink-Based Inter Satellite Link(ISL)across Low Earth Orbit(LEO)[C] / / 2022 2ndInternational Conference on Power Electronics&IoT Applications in RenewableEnergy and its Control(PARC).IEEE,2022:1-5.

[0181]

[24] Zhang H,Liu R,Kaushik A,et al.Satellite edge computing withcollaborative computation offloading:An intelligent deep deterministic policygradient approach[J].IEEE Internet of Things Journal,2023,10(10):9092-9107.

[0182]

[25] Gao X,Liu R,Kaushik A.Virtual network function placement insatellite edge computing with a potential game approach[J].IEEE Transactionson Network and Service Management,2022,19(2):1243-1259.

[0183]

[26] Gao X,Liu R,Kaushik A,et al.Dynamic resource allocation forvirtual network function placement in satellite edge clouds[J].IEEETransactions on Network Science and Engineering,2022,9(4):2252-2265.

[0184]

[27] Li Q,Wang S,Ma X,et al.Service coverage for satellite edgecomputing[J].IEEE Internet of Things Journal,2021,9(1):695-705.

[0185]

[28] Cao X,Yang B,Shen Y,et al.Edge-assisted multi-layer offloadingoptimization of LEO satellite-terrestrial integrated networks[J].IEEE Journalon Selected Areas in Communications,2022,41(2):381-398.

[0186]

[29] Wu Y C,Dinh T Q,Fu Y,et al.A hybrid DQN and optimization approachfor strategy and resource allocation in MEC networks[J].IEEE Transactions onWireless Communications,2021,20(7):4282-42.

Claims

1. A method for offloading aircraft tasks based on DDRL in a low-orbit satellite network, characterized by the following steps: include: (1) Build the LEO low-orbit satellite network system into a LEO satellite mobile edge computing network system; (ii) Model the utility maximization problem of the LEO satellite network system as a joint decision-making problem of optimizing task offloading and resource allocation; (3) The joint decision-making problem is transformed into a Markov decision process (MDP). Under the framework of dual deep reinforcement learning (DDRL), the dual deep Q network algorithm (DDQN) is used to learn the optimal task offloading decision. On this basis, the time-difference triple policy gradient algorithm (TD3PG) is used to obtain the optimal resource allocation strategy.

2. The method for offloading aircraft tasks based on DDRL in a low-orbit satellite network according to claim 1, characterized in that In step (a), the LEO satellite constellation is divided into multiple management domains; each management domain is a subnetwork system; any management domain consists of a cluster head satellite and S cluster member satellites equipped with mobile edge computing MEC servers, and the cluster member satellites are represented by a collection Indicates that the member satellites There are inter-satellite links (ISLs) between the cluster head satellite and member satellites, and between some member satellites. Aircraft User Collection Aircraft users The number of aircraft users is represented by U. In the following text, "aircraft users" are referred to as "users". It is assumed that the users are located in remote areas and can only obtain services by accessing the LEO satellite mobile edge computing network. When the users enter a management domain, they automatically access the cluster head satellite, which offloads the tasks to the member satellites for processing according to the types of tasks to be offloaded by the users. The LEO satellite network system operates in time slot mode, and the set of time slots is represented by Indicates that the time slot Time is divided into Y time slots of equal length, each time slot length is τ; Assuming that the LEO satellite constellation has a stable topology and connectivity within a certain period of time, at the beginning of each time slot, the aircraft randomly and parallelizes the generation of tasks, which are independent and indivisible; assuming that the time interval between task arrivals follows an exponential distribution with a mean value of μ, in time slot t, the task generated by user u in time slot t is represented by the triple Indicates that, where l u,t and c u,t Represent the data size and workload of the task respectively, is the maximum tolerable delay of the task; the total number of tasks uploaded by users during the operation of the LEO satellite network system is represented by K; The task offloading and resource allocation process in the LEO satellite network system includes: first, the user accesses and transmits the task to the cluster head satellite for queuing; then, the cluster head satellite unloads the task from the task waiting queue to the member satellite; finally, the task is processed on the member satellite; where: When the cluster head satellite receives a user task, it puts the task to be unloaded into the unloading queue N(t); the decision matrix symbol for the cluster head satellite to unload the task to the member satellite is Indicates that the matrix Indicates whether to offload the task of the i-th user to the j-th member satellite for processing. Indicates that the task of the i-th user is offloaded to the j-th member satellite for processing. It means that the task of the i-th user will not be offloaded to the j-th member satellite for processing; When the task is unloaded from the cluster head satellite to the member satellite s, s is the unloaded task. Allocate computing resources; set tasks The computing resource allocation strategy is used It means that it is a task The computing resources provided are And there is Indicates the maximum computing resources possessed by a member satellite s; (Once a task is offloaded to a member satellite, the orthogonal computing resources allocated for the task will be retained and occupied for multiple time periods until the task calculation is completed) The computing resources allocated to the member satellite in time slot t cannot exceed the current available resources, which can be expressed as Among them, φ s,t Indicates the resources of satellite s that have been occupied in time slot t.

3. The method for offloading aircraft tasks based on DDRL in a low-orbit satellite network according to claim 2, characterized in that In the LEO low-orbit satellite network system, a. The communication between the user and the cluster head satellite is: The user uploads the task through the link with the cluster head satellite; the cluster head satellite receives the signal y from user u in time slot t. u,t Expressed as Among them, p u and u Represent the transmission power and data signal of user u respectively; N0~(0,σ 2 ) is the additive white Gaussian noise AWGN of the channel, σ 2 is the noise variance; h u,t represents the AWGN channel between user u and the cluster head satellite in time slot t, which is expressed as h u,t =d u or u (d u,t ) -β (3) Among them, δ u ~(0,1) is a complex Gaussian variable with Rayleigh fading, η u is the shadow fading following a log-normal distribution, β is the path loss exponent; d u,t is the actual distance between user u and the cluster head satellite in time slot t; Assume that the user can only start transmitting requests at an elevation angle α, where α is the minimum angle symmetrical to the line connecting the center of the earth and the user's position. Let θ represent the central angle corresponding to the arc covered by the satellite, R is the radius of the earth, H is the satellite orbit height, and α is known. The relationship between θ and α is expressed as Therefore, the actual distance between the cluster head satellite and user u at time slot t is Assuming that the satellite spectrum is orthogonally allocated to all users, and the total communication bandwidth of the cluster head satellite is evenly distributed among the currently associated users, according to the Shannon formula, the data transmission rate from user u to the cluster head satellite is expressed as: Where B represents the total bandwidth of the user and satellite transmission link frequency band; the worse channel h u,t and a larger number of associated users U will result in a slower data transmission rate; Considering the delay problem, let c represent the speed of light, then the time delay between user u and the cluster head satellite at time slot t is At the same time, the task The transmission energy consumption from user u to the cluster head satellite is b. The communication between satellites is: N(t) is used to store the tasks uploaded to the cluster head satellite in time slot t but not yet unloaded; the unloading queue N(t+1) in the next time slot is obtained by subtracting the amount of unloaded task data B(t) and adding the amount of task data newly uploaded to the cluster head satellite get: in, Represents the relationship between time t and task The associated task arrival rate follows a Poisson distribution, and τ is the length of a time slot; Assuming that the CPU frequency of the cluster head satellite is f′ when processing the unloading queue in time slot t, the amount of task data processed by the cluster head satellite in the unloading queue in one time slot is Where J is the number of CPU cycles required to process each bit of task data. Therefore, the queue delay of the unload queue N(t) is expressed as Among them, X N(t) Indicates the queue length, It represents the average task arrival rate of queue N(t), which can be expressed as At the same time, energy consumption is not considered during task queuing; Assume that the task is only unloaded once from the cluster head satellite to the member satellite s; after the cluster head satellite makes the unloading decision, the task It will be unloaded from the unloading queue of the cluster head satellite to the member satellite for processing; then the time delay of user u's task from the cluster head satellite to the unloading member satellite in time slot t is expressed as in, represents the dynamic distance between the member satellite s and the cluster head satellite in time slot t. At the same time, let p be the transmission power of the cluster head satellite, then the task The transmission energy consumption is expressed as In time slot t, the computing resources allocated to user u by unloading satellite s are Then process the task The calculation time is Execute tasks on uninstalled satellites The computational energy consumption is expressed as 4. The method for offloading aircraft tasks based on DDRL in a low-orbit satellite network according to claim 3, characterized in that In step (ii), Task The total delay from access to unloading is T u,t It is composed of (7), (11), (13) and (15) and is expressed as And there is That is, the end-to-end delay of each task does not exceed the maximum tolerable delay. This condition is used as a sign of whether the task is completed or not, and a binary variable a is used to indicate whether the task is completed or not. u,t It means that If the delay constraint is met, the task is considered to be successfully processed. u,t =1, otherwise the task is considered failed. u,t =0; The system energy consumption generated in this process is composed of (8), (14) and (16), which can be expressed as Then, the task The end-to-end latency and system energy consumption are determined by the joint strategy of task offloading and resource allocation; At the beginning of each time slot, the cluster head satellite decides which member satellite to offload the task to and how much computing resources to allocate for the task; The system cost includes the sum of the delay and energy consumption of all tasks during the system operation. The joint optimization of task offloading and resource allocation is described as the system cost minimization problem, which is expressed as Among them, ξ∈[0,1] is used to weigh the proportion of energy consumption cost and delay cost in the system cost. The problem of minimizing system cost is replaced by the problem of maximizing system utility P1, which is modeled as Among them, λ is the overhead factor, which is used to solve the problem that the task delay and energy consumption cannot be taken into account due to unfinished tasks. λ is a constant and sufficiently large. Constraints (21a) and (21b) indicate that each user can offload at most one task to one LEO satellite. (21c) indicates that the resources allocated by the satellite to the task shall not exceed the maximum resources owned by the satellite. (21d) indicates that the sum of the computing resources allocated by each satellite to the current offloaded task shall not exceed the current available resources of the satellite.

5. The method for offloading aircraft tasks based on DDRL in a low-orbit satellite network according to claim 4, characterized in that In step (iii), the problem transformation method is: Due to the correlation of decisions in adjacent time slots, the task offloading and resource allocation process is transformed into an MDP problem; MDP consists of five elements: state space S, action space A, state transition probability P, reward function R, and discount factor γ; at time slot t, First, the cluster head satellite obtains the member satellites’ status information about load conditions and remaining resources, as well as uploaded user mission information; Then, the cluster head satellite outputs the task offloading and resource allocation decision, and returns the resource allocation decision to the member satellites while making the offloading action. The member satellites allocate resources for task processing according to the allocation decision. Finally, the member satellites return the results to the users; The cluster head satellite learns and implements strategies through continuous interaction with the system environment and obtains rewards to improve the strategy.

6. The method for offloading aircraft tasks based on DDRL in a low-orbit satellite network according to claim 5, characterized in that For DDRL training and optimization: The DDRL framework consists of two modules: DDQN and TD3PG. DDQN is used for offloading decision problems, and TD3PG is used for computing resource allocation problems. The interaction process between user tasks and system environment includes: Step ①: DDQN takes the environment state as input and iteratively provides services to each user; Step ②: DDQN processes the information and outputs appropriate offloading decisions for each user; Step ③: The environmental state and offloading decision are processed as input by the TD3PG algorithm; Step ④: TD3PG gives appropriate computing resource allocation; Step ⑤: DDRL outputs the final joint decision; In each interaction process, first, the task offloading action is performed through DDQN Then, perform resource allocation actions according to TD3PG Task From state S t Action A u,t Enter state S t+1 The reward value r(S t ,A t ,S t+1 ) with the symbol r u,t Represents; when DDQN interacts with the environment, it follows the ε-greedy strategy to perform the action of the offloading decision; ε is a probabilistic decision factor ε, which is between 0 and 1; In the DDQN learning process, ε is used to represent the probability of randomly selecting an action, and 1-ε is used to represent the probability of executing the action given by the DDQN main network. probability; when just starting to explore the system environment, set a larger value for ε; as learning accumulates, the value of ε gradually decreases, and the agent will make more use of the knowledge it has learned to choose the action that seems to have the greatest reward; During the TD3PG learning process, since TD3PG is suitable for continuous actions Then it adds random noise when exploring the system environment. The noise obeys a normal distribution with variance σ, and the value of σ gradually decreases during the learning process; After the interaction is completed, an observation for DDQN and an observation for TD3PG are generated, which are stored in independent replay buffers B1 and B2 respectively; When task offloading and resource allocation are based on DDRL, First, update DDQN: Set up the Q network and target network DDQN collects quadruplets from replay buffer B1 For Q and learning; when using samples to update Q, DDQN Calculate the value y u,t As the target value, it is expressed as Where γ is the decay factor; the loss function for updating Q is the square error function, defined as Next, the gradient is calculated and the Q network is updated every g1 steps. At the same time, by setting Update every G1 step Then, update TD3PG: TD3PG consists of two independent value networks C1 and C2, a strategy network A and its corresponding target network. as well as Actor Network A predicts resource allocation strategy Critc networks C1 and C2 determine the q-value of the strategy generated by the Actor network during the learning process; Before TD3PG training, some values ​​are sampled from the playback buffer B2; during training, some values ​​are sampled from the target network and Calculate the target value in Then, the target value y u,t Compared with the q value, the difference between the two is the time difference TD error δ; this function is a square error, defined as The gradient of the above formula is used to update the Critic network C1 and C2; To update the Actor network A, calculate the gradient of the value q1=C1(s,a) predicted by the Critic network C1; use the gradient ascent method, i.e., minimize -q1 to maximize the value of q1; train the networks A, C1, and C2 to update every g2 steps; at the same time, the target network and The strategy network is updated every G2 steps and the value network is updated every μ times.

7. The method for offloading aircraft tasks based on DDRL in a low-orbit satellite network according to claim 5, characterized in that The key elements of MDP are described as follows: State space: At time slot t, the cluster head satellite has the state information of all aircraft users and member satellites from a holistic perspective; the system state is expressed as L u,t represents the geographical location of user u at time slot t. The location information is described as a two-dimensional vector, including the longitude and latitude in the geodetic coordinate system, and the elevation is set to 0; D u,t is a U-dimensional vector, each element of which is the distance between user u and the cluster head satellite at time slot t; Represents the attribute information of the task uploaded by user u in time slot t; is an S-dimensional vector, which refers to the distance between satellite s and the cluster head satellite at time slot t; φ s,t is an S-dimensional vector, indicating the number of resources currently occupied by each member satellite; N(t) represents the queue status of the current unloading queue; Action space: The action refers to the selection of unloading satellites and resource allocation scheme made by the cluster head satellite in time slot t, which is defined as in and They represent task offloading and resource allocation respectively. In the process of dynamic task offloading and resource allocation, the cluster head satellite needs to observe the system environment S t Determine the satellite to be unloaded and make an unloading decision based on the system environment observation information S t , determine the unloading of satellite as the task Allocated computing resources; Reward: In system state S t Perform tasks for users Take action A u，t After that, the system environment enters a new state S t+1 And return the corresponding reward; put the system in state S t The total cost under t ) indicates that when the system undergoes action A u,t Enter state S t+1 After that, state S t and S t+1 There is a cost difference ψ(S t )-ψ(S t+1 ); then, the design of the reward function is based on the cost difference of the system state transition process, expressed as r(S t ,A t ,S t+1 )=ψ(S t )-ψ(S t+1 ) When r(S t ,A t ,S t+1 )>0, that is, when the system cost decreases with time slots, the action gets a positive reward value, otherwise the reward value is negative, indicating that the action is punished.

8. The method for offloading aircraft tasks based on DDRL in a low-orbit satellite network according to claim 2, characterized in that In the management domain of the LEO satellite network system, cluster head satellites are regularly elected; In the LEO satellite network system, if a user is covered by multiple management domains, the cluster head satellite of the management domain with high channel quality is selected by default for access.

Images