Method for aircraft task offloading based on ddrl in low earth orbit satellite network

Abstract

A low-orbit satellite network based on DDRL aircraft task offloading method, the steps of which include: (1) constructing a LEO low-orbit satellite network system into a LEO satellite mobile edge computing network system; (2) modeling the LEO low-orbit satellite network system utility maximization problem as a joint decision problem of optimal task offloading and resource allocation; (3) converting the joint decision problem into a Markov decision process MDP, and using a double deep Q network algorithm DDQN to learn the optimal task offloading decision under a double deep reinforcement learning DDRL framework, and using a time difference triple policy gradient algorithm TD3PG to obtain the optimal resource allocation strategy. Simulation experiments show that compared with the benchmark algorithm, the scheme can effectively access and coordinate concurrent tasks, and has better convergence and superiority under different environmental variables.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of satellite communication, and particularly relates to an aircraft task offloading method based on double deep reinforcement learning (DDRL) in a low-orbit satellite network. BACKGROUND

[0002] Low-orbit (Low Earth Orbit, LEO) satellite communication has the advantages of long transmission distance, low time delay and cost, and no restriction by topography, and can effectively solve the "air route information island" problem of civil aircraft such as civil passenger aircraft during flight. With the increasing amount of business data, the central processing calculation time delay of the traditional satellite network in returning data to the ground processing center is increasing, which is in conflict with the low time delay requirement of the business. The emergence of edge computing provides a new mode for users to use the computing power of the network edge node for data processing. In the edge computing mode, each satellite is regarded as an edge node, and on-orbit edge computing is realized, and data can be processed at the data source, so that the delay caused by satellite data transmission can be reduced.

[0003] Under the above background, documents [3], [4], [5] and [6] have researched the satellite-based edge computing from the architecture and application, and proposed a satellite edge computing architecture, which divides the satellite edge computing system into three parts including a user node, a satellite edge node and a ground data center, so that the business time delay can be reduced and the bandwidth can be saved. Wang et al. [7] further introduced the corresponding satellite hardware structure and software architecture for changing the traditional satellite into a space edge computing node, and through the virtualization of satellite resources, flexible computing services are realized. Xie et al. [8] comprehensively and elaborately analyzed the cooperative computing offloading, multi-node task scheduling and mobility management in the integrated satellite-ground edge computing network, and pointed out the technical challenges faced by the satellite edge computing. Li et al. [9] considered that the satellite is an edge computing platform with computing and storage resources, and solved the system service request scheduling decision and service placement problem by solving the mixed integer linear programming problem. Cao et al.

[10] proposed a satellite edge cloud architecture based on software-defined network and network function virtualization, and proposed a computing resource scheduling method considering energy consumption, service delay and resource utilization. Since the communication, power consumption and computing resources of the satellite platform are limited, documents

[11] ,

[12] ,

[13] and

[14] research offloading algorithms for joint computing and communication resource allocation and offloading decisions, and realize offloading decisions based on deep reinforcement learning, game theory and heuristic search algorithm respectively, so as to reduce the system service time delay and power consumption.

[0004] However, low-orbit satellite networks still face challenges. Although deploying a mobile edge computing (MEC) server on a LEO satellite greatly enhances its computing power, limited by the volume and energy supply of the satellite, the computing resources that each satellite can provide are still limited. By reasonably arranging the computing resources within the satellite constellation, tasks can be transferred to satellites with more idle resources for processing. By efficiently utilizing limited satellite resources, not only can the requirements of users for low latency be met, but the workload of individual satellites can also be reduced, and energy consumption can also be reduced. Therefore, how to reasonably allocate and schedule the satellite resources dispersed at the network edge to achieve direct offloading of aircraft tasks to low-orbit satellites for processing and reduce the time delay and energy consumption in the task transmission process has become one of the challenges of satellite edge computing. SUMMARY

[0005] In view of the above challenges, the present application designs a task offloading and resource allocation joint optimization scheme based on a double deep reinforcement learning (DDRL) framework, aiming to maximize the system utility of the LEO satellite network.

[0006] The aircraft task offloading method based on DDRL in the low-orbit satellite network of the present application includes the following steps:

[0007] (I) The LEO low-orbit satellite network system is constructed into a LEO satellite mobile edge computing network system;

[0008] (II) The problem of maximizing the system utility of the LEO low-orbit satellite network is modeled as a joint decision problem of optimizing task offloading and resource allocation;

[0009] (III) The joint decision problem is converted into a Markov decision process (MDP), and under the double deep reinforcement learning (DDRL) framework, the double deep Q network algorithm (DDQN) is used to learn the optimal task offloading decision, and on this basis, the time difference triple policy gradient algorithm (TD3PG) is used to obtain the optimal resource allocation strategy.

[0010] The main contributions of the present application are as follows:

[0011] (1) For the task offloading and resource allocation problem in the LEO satellite network, considering the mobility of low-orbit satellites, the difference in satellite computing power, multiple user tasks and other factors, the present application proposes a task offloading and resource allocation model with the goal of maximizing system utility.

[0012] (2) Considering that decision offloading and resource allocation is a complex optimization problem with mixed discrete and continuous variables, the present application uses a double deep Q network (DDQN) algorithm to learn the system environment and obtain the optimal decision offloading under the environment, and then uses a TD3PG algorithm to optimally allocate resources.

[0013] (3) The present application simulates a LEO satellite network environment on a satellite environment simulation platform to verify the performance of DDRL. Simulation experiments show that, compared with the benchmark algorithm, the scheme can effectively access and cooperatively compute concurrent tasks, and has better convergence and superiority under different environmental variables. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 represents a LEO satellite network system;

[0015] Figure 2 represents a task offloading and resource allocation model;

[0016] Figure 3 represents the geometric relationship between satellites and users;

[0017] Figure 4 represents a DDQN and TD3PG cooperative training framework;

[0018] Figure 5 represents the convergence curves of different algorithms;

[0019] Figures 6(a) to 6(c) respectively represent the performance of different algorithms under different communication resources, wherein:

[0020] Fig. 6(a) represents the reward value, Fig. 6(b) represents the system utility, and Fig. 6(c) represents the task success rate;

[0021] Figures 7(a) to 7(c) respectively represent the performance of different algorithms under different computing resources, wherein:

[0022] Fig. 7(a) represents the reward value, Fig. 7(b) represents the system average energy consumption, Fig. 7(c) represents the system average time delay, and Fig. 7(d) represents the task success rate;

[0023] Figures 8(a) to 8(b) respectively represent the convergence curves and probability densities, wherein:

[0024] Fig. 8(a) represents the convergence curves of the three algorithms, and Fig. 8(b) represents the system utility probability density. DETAILED DESCRIPTION

[0025] 1. SUMMARY

[0026] Low Earth Orbit (LEO) satellite communication has the advantages of long transmission distance, wide coverage, and no restriction by terrain, and has become an important communication means for civil aviation and general aviation industry. However, the low-orbit satellite network is a highly heterogeneous and dynamic environment, and the mobility of satellite nodes, the complexity of communication links, the uneven distribution of aircraft in space and time, and the coexistence of various services make task offloading and resource allocation face many challenging problems.

[0027] Therefore, the present application proposes an aircraft task offloading method based on double deep reinforcement learning (DDRL), aiming to maximize the overall system utility.

[0028] Firstly, the system utility maximization problem is modeled as a joint optimization problem of task offloading and resource allocation, considering the computing capacity and coverage time of LEO satellites.

[0029] Next, the problem is converted into a Markov decision process, and the double deep Q network algorithm DDQN is used to learn the optimal task offloading decision, and on this basis, the temporal difference triple policy gradient algorithm (TD3PG) is used to obtain the optimal resource allocation strategy.

[0030] Simulation experiments show that under different computing resources and communication resources, the scheme proposed in the present application is superior to other benchmark schemes in system utility, proving the availability of the proposed framework.

[0031] 2. System model and problem modeling

[0032] This subsection first describes the network model of the LEO satellite network system. Secondly, the communication model and the computing model of the task offloading and resource allocation based on the association between the aircraft user and the satellite are established. Finally, the optimization problem of minimizing the system energy consumption under the constraint condition is proposed.

[0033] The key symbols used later and the related descriptions are listed in Table 1.

[0034] Table 1 Important symbols

[0035]

[0036] 2.1 Network model

[0037] In the following, the network model of the LEO satellite network system is described. Figure 1In the illustrated LEO satellite network system architecture, the LEO satellite constellation (a satellite constellation refers to a group of satellites with a specific working relationship with each other, which work together to complete a specific space mission) is divided into several management domains, and each management domain is a sub-network system. The management domain is composed of 1 cluster head satellite and S cluster member satellites carrying MEC servers, and the cluster member satellites are represented by the set , where the member satellites There are inter-satellite links ISL (Inter Satellite Link) between the cluster head satellite and the member satellites and between some member satellites. The fixed satellite management node faces energy loss, spatial position change and other situations, and there are network supplement, function update and other situations in the later period. The current cluster head satellite control management capability fluctuates, and periodic cluster head election is needed to maximize the control capability of the cluster head satellite in the system. Therefore, the present application adopts the multi-factor weighted cluster head election method of "static index + dynamic index" in document

[15] to regularly elect the cluster head satellite. The aircraft users are represented by the set , taking civil aircraft as an example, where the aircraft users The number of aircraft users is represented by U. Assuming that the aircraft is located in a remote area at this time, the aircraft can only access the LEO satellite mobile edge computing network to obtain services

[16] . When the aircraft enters a LEO satellite constellation management domain, it will automatically access the cluster head satellite of the management domain, and the cluster head satellite will unload it to the member satellites for processing according to the type of task. It should be noted that if the aircraft is covered by multiple management domains, the cluster head satellite of the management domain with high channel quality is selected by default.

[0038] The LEO satellite network system operates in time slot mode, and the time slot is represented by the set , and the time slot The time is divided into Y time slots of equal length, and each time slot has a length of τ. It is assumed that the routing table and trajectory information of the LEO constellation are known to the cluster head satellite. The link state between satellites usually depends on the relative position between the satellite and the earth

[17] , and when the satellite is located in the non-polar region of the earth, the link state of the ISL is relatively stable. Accordingly, in order to simplify the model, we assume that the LEO satellite constellation has stable topology and connectivity within a certain time

[18] . At the beginning of each time slot, the aircraft will randomly and in parallel generate tasks, and these tasks are independent and indivisible. It is assumed that the time interval of task arrival follows an exponential distribution with an average value of μ. In time slot t, the task generated by user u in time slot t is represented by the triple , where, l u,t and c u,t represent the data size and workload of the task, is the maximum tolerable latency of a task. In addition, the total number of tasks uploaded by users during system operation is denoted by K. The output data of a task is usually much smaller than the input data

[19] , so the delay of returning the computation result can be ignored. In addition, assuming that the user identity, communication protocol, and other information are usually sent as part of the task allocation message when the cluster head satellite offloads tasks to member satellites, after completing the processing of several time slots, the member satellites can be configured according to this information to establish a communication link with the user and then return the processing result.

[0039] As shown in Figure 2 , the model of task offloading and resource allocation in the LEO satellite network system includes three steps: aircraft access and transmission of tasks to the cluster head satellite for queuing, offloading of tasks from the waiting queue to the member satellites by the cluster head satellite, and task processing on the member satellites. First, when the cluster head satellite receives a large number of user tasks, the waiting queue during task distribution and offloading needs to be considered. The cluster head satellite places the tasks to be offloaded into the offloading queue N(t). Second, let the decision of the cluster head satellite to offload tasks to member satellites be denoted by the matrix symbol . The matrix indicates whether the task of the i-th user is offloaded to the j-th satellite for processing, indicates that the task of the i-th user is offloaded to the j-th satellite for processing, indicates that the task of the i-th user is not offloaded to the j-th satellite for processing. Finally, after the task is offloaded to the member satellite s by the cluster head satellite, s needs to allocate computing resources to the offloaded task . Let the computing resource allocation strategy of the task be denoted by . The size of the computing resource provided by to the task is and has denotes the maximum computing resource possessed by a member satellite s. In addition, the present application adopts a reservation-based resource allocation mechanism

[20] . After a task is offloaded to a satellite, the orthogonal computing resource allocated to the task will be reserved and occupied for several time periods until the task computation is completed, which can ensure that the task is immediately processed when it arrives at the satellite and there is no additional delay jitter

[21] . Therefore, due to the different processing processes of tasks on different satellites, the available computing resources of the satellite for the current task are also different. The computing resources allocated by the satellite at time slot t cannot exceed the current available resources, which can be expressed as

[0040]

[0041] where φ s,t denotes the resources already occupied by the satellite s at time slot t.

[0042] 2.2 Communication Model

[0043] (1) Satellite-Aircraft Communication Model

[0044] The user uploads the task through the link between the cluster head satellite and the user. The transmission link uses the Ka band. Here, the application considers a quasi-static fading channel model in a time slot according to the literature

[22] . Specifically, the signal y received by the cluster head satellite from the user u at time slot t is u,t

[0045]

[0046] where p u and s u represent the transmit power and data signal of user u, respectively. N0~(0,σ 2 ) is the Additive White Gaussian Noise (AWGN) of the channel, and σ 2 is the noise variance. h u,t represents the AWGN channel between user u and the cluster head satellite at time slot t, which is represented as

[0047] h u,t = δ u η u (d u,t ) -β (3)

[0048] where δ u ~(0,1) is a complex Gaussian variable with Rayleigh fading, and η u is a shadow fading that follows a lognormal distribution. d u,t is the actual distance between user u and the cluster head satellite at time slot t, which is obtained from the geometric relationship between user u and the cluster head satellite, as shown in Figure 3 Since there is a difference of two or more orders of magnitude between the height of the low-orbit satellite and the height of the general aircraft, this paper does not consider the height of the aircraft, and for the same reason, the speed of the aircraft is not considered. It is assumed that the user can only start transmission request at a certain elevation angle a, which is the smallest angle symmetric to the line connecting the center of the earth and the user's location. Let θ represent the central angle corresponding to the radian of the satellite coverage, R be the radius of the earth, and H be the satellite orbit height. Under the premise of ignoring other factors, assuming that a is known, the relationship between θ and a can be represented as

[0049]

[0050] Therefore, the actual distance between the cluster head satellite and user u at time slot t is

[0051] ​

[0052] In addition, β is the path loss exponent, which means that the communication condition between the satellite and the user is negatively correlated with the distance between the two.

[0053] To simplify the problem, it is assumed that the satellite spectrum is orthogonally allocated to all users, and the total communication bandwidth of the cluster head satellite is evenly distributed among the currently associated users. Therefore, according to the Shannon formula, the data transmission rate from user u to the cluster head satellite can be expressed as:

[0054]

[0055] Where B represents the total bandwidth of the Ka band. Obviously, the worse channel h u,t A larger number of associated users U will result in a slower data transmission rate.

[0056] Considering the delay problem, there is a communication delay inside the machine when the data packet is pushed to the link, and there is a propagation delay outside the machine when the electronic signal is transmitted. Let c represent the speed of light, then the time delay between user u and the cluster head satellite at time slot t is

[0057]

[0058] In addition, the task

[0059] The transmission energy consumption from user u to the cluster head satellite is

[0060]

[0061] (2) Inter-satellite communication model

[0062] N(t) is used to store the tasks that have been uploaded to the cluster head satellite in time slot t but have not been unloaded. The unloading queue N(t+1) in the next time slot is obtained by subtracting the amount of unloaded task data B(t) and adding the amount of task data newly uploaded to the cluster head satellite. get:

[0063]

[0064] in, Represents the relationship between time t and task The related task arrival rate follows the Poisson distribution, and τ is the length of a time slot. Assuming that the CPU frequency of the cluster head satellite is f′ when processing the unloading queue in time slot t, the amount of task data processed by the cluster head satellite in the unloading queue in one time slot is

[0065]

[0066] where J represents the number of CPU cycles required to process one bit of task data. Thus, the queue delay of the offloading queue N(t) can be expressed as

[23]

[0067]

[0068] where X N(t) represents the queue length, represents the average task arrival rate of the queue N(t), which can be expressed as

[0069]

[0070] In addition, the energy consumption is not considered in the task queuing process.

[0071] To improve real-time performance, it is assumed that the task can only be offloaded from the cluster head satellite to the member satellite s once, and there is no multi-hop offloading process between the member satellites. After the cluster head satellite makes the offloading decision, the task will be offloaded from the offloading queue of the cluster head satellite to the member satellite for processing. Therefore, the time delay of the task of user u from the cluster head satellite to the offloading satellite at time slot t is represented as

[0072]

[0073] where, represents the dynamic distance between the member satellite s and the cluster head satellite at time slot t. At the same time, let p be the transmission power of the cluster head satellite, then the transmission energy consumption of the task can be represented as

[0074]

[0075] 2.3 Calculation model

[0076] In time slot t, the computing resource allocated to user u by offloading satellite s is Therefore, the computing time for processing task is

[0077]

[0078] In this phase, the task computing delay is mainly considered, and the queuing delay before the current task computing is ignored.

[0079] The present application uses a widely used energy consumption model based on one computing cycle, and the processing energy consumption of one task on the satellite is calculated as e = κf 2

[24] , where κ is the energy consumption coefficient, which depends on the effective switching capacitance of the chip architecture, and f is the CPU frequency. Then, the computing energy consumption of executing task on offloading satellite s is represented as

[0080]

[0081] 2.4 Problem modeling

[0082] Based on the above discussion, the task The total latency T from access to offload u,t is composed of (7), (11), (13) and (15), denoted as

[0083]

[0084] and has That is, the end-to-end delay of each task cannot exceed the maximum tolerable latency, and this condition is taken as the flag of whether the task is completed or not, denoted by binary variable a u,t , defined as

[0085]

[0086] If the delay constraint can be met, the task is considered to be successfully processed, and at this time a u,t = 1, otherwise the task is considered to fail, and at this time a u,t = 0.

[0087] The system energy consumption generated in this process is composed of (8), (14) and (16), denoted as

[0088]

[0089] Therefore, the end-to-end delay of the task and the system energy consumption are determined by the joint strategy of task offloading and resource allocation. At the beginning of each time slot, the cluster head satellite decides which member satellite to offload the task to and how much computing resource to allocate for the task.

[0090] The system cost includes the sum of the delay and energy consumption of all tasks in the system during operation, and the joint optimization of task offloading and resource allocation can be described as the minimization problem of the system cost, denoted as

[0091]

[0092] Wherein, ξ ∈ [0, 1] is used to weigh the proportion of energy consumption cost and delay cost in the system cost.

[0093] The minimization of the system cost is replaced by the maximization problem of the system utility, modeled as

[0094]

[0095] where λ is an overhead factor to solve the problem that the task delay and energy consumption cannot be counted due to the unfinished task, and λ is a constant and sufficiently large. Constraints (21a) and (21b) indicate that each user can offload at most one task to a LEO satellite; (21c) represents that the resources allocated to the task by the satellite cannot exceed the maximum resources possessed by the satellite; (21d) represents that the sum of the computing resources allocated to the current offloaded tasks by each satellite cannot exceed the current available resources of the satellite.

[0096] 3. Algorithm design

[0097] According to the above discussion, is a discrete variable, is a continuous variable, and the problem to be solved by the present application is a complex optimization problem involving a discrete variable and a continuous variable. Therefore, a DDRL framework is proposed in this section to solve the joint decision problem. For the discrete variable environment, DDQN is used to learn the system environment to obtain the optimal decision offloading under the environment, and then for the continuous variable environment, TD3PG is used to allocate resources to obtain the optimal resource allocation strategy. The specific process is described as follows.

[0098] 3.1 Problem transformation

[0099] Considering the correlation between decision offloading and resource allocation, i.e., resource allocation and task offloading are coupled, this paper jointly processes the two problems.

[0100] Due to the correlation of adjacent time slot decisions, the task offloading and resource allocation process in the above dynamic scenario can be transformed into an MDP problem. MDP consists of five elements: state space S, action space A, state transition probability P, reward function R, and discount factor γ. At time slot t, the cluster head satellite can obtain the state information of the member satellites, such as load, remaining resources, etc., and the uploaded user task information. Then the cluster head satellite outputs the task offloading and resource allocation decision, and returns the resource allocation decision to the member satellite while making the offloading action, and the member satellite allocates resources for task processing. Finally, the member satellite returns the result to the user. The cluster head satellite learns and implements the strategy through continuous interaction with the system environment, and obtains the reward to improve the strategy. The description of the key elements in MDP is as follows:

[0101] State space: At time slot t, the cluster head satellite masters the state information of all aircraft users and cluster member satellites from the overall perspective. The system state is represented as L u,t represents the geographical location of user u at time slot t, and the position information is described as a two-dimensional vector containing the longitude and latitude in the geodetic coordinate system, and the elevation is set to 0. u,t is a U-dimensional vector, and each element is the distance between user u and the cluster head satellite at time slot t. denotes the attribute information of task of user u at time slot t. is an S-dimensional vector, which denotes the distance between satellite s and cluster head satellite at time slot t. φ s,t is also an S-dimensional vector, which denotes the number of resources currently occupied by each member satellite. N(t) denotes the queuing condition of the current offloading queue.

[0102] Action space: Action refers to the selection of offloading satellite and the resource allocation scheme made by cluster head satellite at time slot t, which is defined as wherein and denote the task offloading action and resource allocation action, respectively. In the process of dynamic task offloading and resource allocation, the cluster head satellite needs to observe the system environment S t to determine the offloading satellite and, on the basis of the offloading decision, to determine the computing resources allocated to the task t .

[0103] Reward: Under the system state S t , the user task

[0104] takes action A u,t , the system environment will enter a new state S t+1 and return the corresponding reward. The total cost of the system under state S t is denoted by the symbol ψ(S t ). When the system enters state S t+1 after action A u,t , there is a cost difference ψ(S t )-ψ(S t+1 ) between states S t and S t+1 . Therefore, the design of the reward function is based on the cost difference in the state transition process, and the specific formula is

[0105] r(S t ,A t ,S t+1 )=ψ(S t )-ψ(S t+1 ) (22)

[0106] When r(S t ,A t ,S t+1 )>0, i.e., the system cost decreases with time slot, the action will get a positive reward value, otherwise the reward value is negative, indicating that the action is punished.

[0107] 3.2 Model training ​

[0108] The DDRL framework contains two modules, DDQN and TD3PG. DDQN is used to solve the offloading decision problem, and TD3PG is used to solve the resource allocation problem. The detailed interaction process between the user task and the environment is shown in Fig. 1. Figure 4 As shown in Fig. 1, the process is divided into five steps:

[0109] Step ①: DDQN takes the environment state as input and iteratively provides services for each user;

[0110] Step ②: DDQN neural network processes information and outputs appropriate offloading decisions for each user;

[0111] Step ③: The environment state and the offloading decisions of the users are taken as input by the TD3PG algorithm for processing;

[0112] Step ④: TD3PG considers the above information and then gives appropriate resource allocation;

[0113] Step ⑤: DDRL outputs the final joint decision.

[0114] Specifically, each interaction first performs task offloading actions through DDQN Then performs resource allocation actions according to TD3PG In addition, the task From state S t After action A u,t , it enters state S t+1 The reward value r(S t , A t , S t+1 ) is represented by the symbol r u,t . When DDQN interacts with the environment, it follows the ε-greedy strategy to perform the action of offloading decisions. ε is a probability decision factor ε, and ε is between 0 and 1. In the DDQN learning process, the probability of randomly selecting an action is represented by ε, and the probability of performing the action given by the DDQN main network is represented by 1-ε .

[0115] When first exploring the environment, a larger value of ε is set, which can bring more randomness and is beneficial to discovering new states and actions. As learning accumulates, the value of ε gradually decreases, and the agent will use more learned knowledge to choose actions that appear to have the highest return. On the other hand, since TD3PG is suitable for continuous action Therefore, its exploration method adds random noise. The noise follows a normal distribution with variance σ, and the value of σ will gradually decrease during the learning process.

[0116] After the interaction is completed, Algorithm 1 generates an observation for DDQN and an observation for TD3PG. These observations are stored in two independent replay buffers B1 and B2, respectively.

[0117] Sample generation of Algorithm 1 task interacting with environment

[0118]

[0119] 3.3 DDRL joint optimization framework

[0120] DDRL-based task offloading and resource allocation is described as Algorithm 2, which samples data from Algorithm 1. Algorithm 2 starts with the update process of the DDQN algorithm. This part sets the Q network and the target network DDQN samples quadruples from replay buffer B1 for learning Q and . When the replay buffer B1 is full, the latest observations will replace the old observations. When updating Q using samples, DDQN calculates the value y from the u,t target value, denoted as

[0121]

[0122] where γ is the decay factor. The loss function for updating Q is the squared error function, defined as

[0123]

[0124] Next, calculate the gradient using the above formula, and update the Q network every g1 steps. At the same time, set update every G1 steps

[0125] The second part of Algorithm 2 is the update process of the TD3PG algorithm. After making offloading decisions for tasks, the algorithm allocates the optimal number of computing resources to the tasks. TD3PG is an Actor-Critic method, which contains two independent value networks and a policy network and their corresponding target networks. The Actor network A predicts the resource allocation policy which is a continuous value of computing resources, and the Critc network C1, C2 judges the q value of the policy generated by the Actor network during the learning process. Before training TD3PG, the algorithm will sample some values from the replay buffer B2. During the training process, the algorithm will calculate the target value from the target networks and

[0126]

[0127] where Then, the target value y u,t The difference between q and y is the time difference (TD) error δ. The function is a squared error defined as

[0128]

[0129] The gradients of the above equation will be used to update the Critic networks C1, C2. To update the Actor network A, the algorithm computes the gradient of the predicted q1 = C1(s, a) value from the Critic network C1. Since the Actor network A is kept fixed while updating the Critic network C1, the algorithm uses the gradient ascent method (minimizing -q1) to maximize the q1 value. The networks A, C1, and C2 are updated every g2 steps. Meanwhile, the target networks and are updated every G2 steps by setting them to be the same as the training networks. In addition, to guarantee the stability of the policy network updates, the policy network is updated only once for every μ value network updates.

[0130] Algorithm 2 Task offloading and resource allocation algorithm based on DDRL

[0131]

[0132]

[0133] 3.4 Time complexity analysis

[0134] First, the total number of training episodes determines the number of iterations the algorithm needs to perform. Each iteration implies a complete training process, including obtaining feedback from the environment and updating the policy. Obviously, more training episodes will directly increase the overall computational load of the algorithm. We denote the size of the total number of training episodes episode as |episode|, then the time complexity is linear with respect to |episode|. Second, the environment termination time Y represents the upper limit of the number of steps the environment runs in each training episode. Each step involves state transition, action selection, and reward calculation processes. With each additional step, the computational load of the algorithm increases accordingly, so the time complexity is also linear with respect to Y. Finally, describe the number of user tasks K. K represents the number of tasks that need to be processed in each training step. In each step, the algorithm needs to calculate and decide for each user task. Therefore, the increase of K will linearly increase the computational load of each step, ultimately affecting the overall time complexity. In summary, the time complexity of DDRL is O(|episode|YK).

[0135] 4. Experimental design and results analysis

[0136] This section uses simulation methods to verify the effectiveness of the proposed method. The method in references

[25]

[26]

[27] is simulated and verified based on the AGI STK (System Tool Kit) software platform. First, a satellite cluster based on the Iridium constellation is set up, and then the users are randomly placed under the coverage of the satellite cluster. Finally, the satellite orbit parameters are set to be closer to the actual scenario. We set the time span of the STK scenario to one hour. The trajectory data samples of the satellite are obtained at a time interval of τ=1.

[0137] In the proposed DDRL-based framework, the DDQN training network Q contains a hidden layer with 256 neurons. The TD3PG training networks A, C1, and C2 also have a hidden layer with 256 neurons each. The target networks of DDQN and TD3PG are and They have the same structure as their corresponding trained networks. In Algorithm 2, we set the number of sampling episodes g1 = g2 = 4, the number of update episodes G1 = 1000, and G2 = 100. Furthermore, the replay buffers B1 = 12000 and B2 = 60000, for a total number of episodes episode = 4000. After the system environment is started, DDQN starts learning first. TD3PG begins learning after its buffer has collected 12000 samples, as TD3PG requires more samples for stable convergence. DDRL is optimized using the Adam optimizer with an initial learning rate of 0.00015. After 1000 episodes, the learning rate is reduced by 90%. Furthermore, the decay factors for DDQN and TD3PG are set to 0.9 and 0.99, respectively. The ε value is initially set to 1.0 and linearly decreased to 0.02. TD3PG uses a normal distribution for exploration, and the variance of the normal distribution is multiplied by 0.99 after each update.

[0138] In addition, the total bandwidth B in the simulation environment is set to 300 MHz, and the transmission power of the user and satellite is p u and 2W and 5W respectively

[28] . The maximum computing resource satellite owned by member satellite s The frequency range is 2 to 4 GHz[5]. The inter-satellite distance and communication capacity are 800 to 1200 km and 10 Gbps, respectively. In addition, the weight parameter ξ in the objective function is set to 0.6. Table 2 summarizes the main parameter settings in the simulation.

[0139] Table 2 Simulation parameters

[0140]

[0141] To objectively evaluate the performance, we select three representative task offloading and resource management strategies for comparison, including:

[0142] Baseline method-1: DQN-DDPG algorithm. Considering that the scenario set in this paper is a complex environment that contains both discrete and continuous action spaces, and DQN and DPG algorithms are classic DRL algorithms that act on discrete action spaces and continuous action spaces, respectively, we combine the two algorithms and compare them with our scheme.

[0143] Baseline method-2: Heuristic algorithm based on greedy offloading strategy

[29] . This strategy follows the principle of the greedy algorithm, which only selects the best decision at the moment without considering the overall optimal solution.

[0144] Baseline method-3: Random method. The task is randomly offloaded to a member satellite, and the resources are randomly allocated, without considering whether the delay and resource constraints are met.

[0145] 4.1 Convergence analysis

[0146] In this subsection, the convergence performance of the four algorithms is evaluated by calculating the average cumulative reward. From Figure 5 It can be observed that the proposed method and baseline method-1 achieve stable task offloading and resource allocation processes during the last training process. Compared with baseline algorithm-1, the proposed method obtains a better cumulative reward value. This is because the DDQN and TD3PG algorithms in the proposed method optimize the overestimation problem in the DQN and DPG algorithms, respectively, which can reduce the oscillation and instability in the learning process. In addition, baseline algorithm-2 and baseline algorithm-3 are not affected by the training process and have poor performance, but baseline algorithm-2 can also find the optimal solution in certain cases, and its performance is relatively better than that of baseline algorithm-3.

[0147] 4.2 Influence of bandwidth and computing resources on performance

[0148] First, Figures 6(a) to 6(c)The performance of each strategy is shown when the satellite has different total bandwidth B. From Fig. 6(a) and Fig. 6(c), it can be observed that when the total bandwidth B is small, the reward rate and task success rate of each algorithm are both low. This is because all aircraft users access the cluster head satellite uniformly and average bandwidth, resulting in data transmission at a small communication rate, increasing transmission delay and energy consumption. As B gradually increases, task data can be transmitted through channels with more communication resources, which can ensure more tasks to be completed on time and increase system utility, as shown in Fig. 6(b) and Fig. 6(c). When B further increases, most tasks can be successfully completed, but the impact on system utility gradually decreases, at which time the reward of each algorithm begins to stabilize. In addition, the performance of the proposed method is optimal compared to the three benchmark algorithms, which means that the task offloading and resource allocation strategy based on DDRL can find the best offloading satellite and optimal resource allocation strategy within a dynamic range of communication bandwidth.

[0149] In addition, this experiment compares the changes in the joint strategy when the satellite has different computing capabilities , as shown in Fig. 7. Figures 7(a) to 7(d) In Fig. 7(a), when is very small (resource scarce), tasks tend to be allocated to inappropriate resources for processing, which cannot be processed within the specified time, resulting in a very large penalty in the learning process. As grows, more tasks can be successfully processed, and the reward will also increase positively. However, when further increases within a certain range, the task success rate will not be significantly improved because the system's demand for has been basically met. On the contrary, the continuously increasing value of will lead to greater computing energy consumption and exponential growth, which leads to a decrease in reward value in the latter half. As can be seen from Fig. 7(b) and Fig. 7(c), because the increase in computing resources will lead to a rapid increase in energy consumption, resulting in an increase in system energy consumption, but a decrease in system latency. At the same time, the method proposed in this paper still maintains the best performance. In addition, as can be seen from Fig. 7(d), the task success rate of each algorithm increases with the increase of , because the improvement of computing energy can bring down the processing delay, thereby improving the task success rate. Among them, the DDRL method proposed in this paper maintains the maximum reward, while the performance of benchmark algorithm-1, benchmark algorithm-2 and benchmark algorithm-3 decreases in turn.

[0150] 4.3 Ablation experiment

[0151] To verify the effectiveness of DDRL, two modified joint optimization algorithms are established for ablation experiments:

[0152] (1) DDQN-DDPG algorithm: using DDPG algorithm instead of TD3PG algorithm in resource allocation decision. DDPG algorithm does not use double Critic architecture and does not use the method of policy delay update.

[0153] (2) DQN-TD3PG algorithm: using DQN algorithm instead of DDQN algorithm in task offloading decision. Specifically, we use the same network for action selection and evaluation, and do not use a separate target network for action value evaluation.

[0154] As shown in FIG. 8(a), the average cumulative reward of DDQN-DDPG and DQN-TD3PG after convergence is 5000 and 4000 respectively, both of which are lower than 7000 of DDQN-TD3PG. This is because when dealing with tasks that require highly accurate value estimation, DDQN and TD3PG usually have higher performance and faster convergence speed than DQN and DPG. In addition, the average cumulative reward of DDQN-DDPG is slightly higher than that of DQN-TD3PG, because the input of continuous action space algorithm needs the output of discrete action space algorithm as a prerequisite, and DDQN has better performance and higher robustness than DQN.

[0155] FIG. 8(b) shows the probability density of system utility distribution of the three algorithms, in which the system utility of DQN-TD3PG, DDQN-DDPG and DDQN-TD3PG is concentrated near 3600, 3800 and 4100 respectively, and the curve width of DDQN-TD3PG is narrower. The mathematical expectation and variance corresponding to the three curves show that DDQN-TD3PG has the optimal joint optimization performance.

[0156] 5. Conclusions

[0157] The present application proposes a DDRL framework based on double deep learning algorithm for task offloading and resource allocation of low-orbit satellite edge computing. The framework contains two DRL algorithms to solve the problem of limited communication and computing resources of aircraft users in the process of task offloading and resource allocation. Through the analysis of the convergence and performance of the proposed scheme under different resources, the effectiveness of the proposed scheme is proved. Finally, the ablation experiment proves that the proposed scheme has better system utility compared with other combinations of DRL. In the next stage, the task can be divided into a certain number of small tasks, and then offloaded to different satellites for distributed processing.

[0158] References

[0159] [1]Mao Y,You C,Zhang J,et al.A survey on mobile edge computing:The communication perspective[J].IEEE communications surveys&tutorials,2017,19(4):2322-2358.

[0160] [2]Denby B,Lucia B.Orbitaledge computing:Machine inference in space[J].IEEE Computer Architecture Letters,2019,18(1):59-62.

[0161] [3]Wei J,Han J,Cao S.Satellite IoT edge intelligent computing:Aresearch on architecture[J].Electronics,2019,8(11):1247.

[0162] [4]Cao S,Zhao Y,Wei J,et al.Space-based cloud-fog computingarchitecture and its applications[C] / / 2019IEEE World Congress on Services(SERVICES).IEEE,2019,2642:166-171.

[0163] [5]Guo J,Du Y.Fog service in space information network:Architecture,use case,security and challenges[J].IEEE Access,2020,8:11104-11115.

[0164] [6] Wei J, Cao S. Application of edge intelligent computing in satellite Internet of Things [C] / / 2019IEEE international conference on smart internet of things (SmartIoT). IEEE, 2019: 85-91.

[0165] [7] Wang Y, Yang J, Guo X, et al. Satellite edge computing for the internet of things in aerospace [J]. Sensors, 2019, 19(20): 4375.

[0166] [8] Xie R, Tang Q, Wang Q, et al. Satellite-terrestrial integrated edge computing networks: Architecture, challenges, and open issues [J]. Ieee Network, 2020, 34(3): 224-231.

[0167] [9] Li C, Zhang Y, Hao X, et al. Jointly optimized request dispatching and service placement for MEC in LEO network [J]. China Communications, 2020, 17(8): 199-208.

[0168]

[10] Cao B, Zhang J, Liu X, et al. Edge–cloud resource scheduling in space–air–ground-integrated networks for internet of vehicles [J]. IEEE Internet of Things Journal, 2021, 9(8): 5765-5772.

[11] Cui G, Long Y, Xu L, et al. Joint offloading and resource allocation for satellite assisted vehicle-to-vehicle communication [J]. IEEE Systems Journal, 2020, 15(3): 3958-3969.

[0169]

[12] Wang B, Feng T, Huang D. A joint computation offloading and resource allocation strategy for LEO satellite edge computing system [C] / / 2020 IEEE 20th International Conference on Communication Technology (ICCT). IEEE, 2020:649-655.

[0170]

[13] Cui G, Li X, Xu L, et al. Latency and energy optimization for MEC enhanced SAT-IoT networks [J]. IEEE Access, 2020, 8: 55915-55926.

[0171]

[14] Tran T X, Pompili D. Joint task offloading and resource allocation for multi-server mobile-edge computing networks [J]. IEEE Transactions on Vehicular Technology, 2018, 68(1): 856-868.

[0172]

[15] Li N, Yue C, Guo H, et al. Large-scale satellite constellation domain management and control strategy design[J]. China Space Science and Technology, 2023, 43(01): 18-28.

[0173]

[16] Casoni M, Grazia C A, Klapez M, et al. Integration of satellite and LTE for disaster recovery[J]. IEEE Communications Magazine, 2015, 53(3): 47-53.

[0174]

[17] Wang F, Jiang D, Qi S, et al. Fine-grained resource management for edge computing satellite networks[C] / / 2019 IEEE Global Communications Conference (GLOBECOM). IEEE, 2019: 1-6.

[0175]

[18] Su Y, Liu Y, Zhou Y, et al. Broadband LEO satellite communications: Architectures and key technologies[J]. IEEE Wireless Communications, 2019, 26(2): 55-61.

[0176]

[19] Tang Q, Fei Z, Li B, et al. Computation offloading in LEO satellite networks with hybrid cloud and edge computing[J]. IEEE Internet of Things Journal, 2021, 8(11): 9169176.

[0177]

[20] Cui G, Long Y, Xu L, et al. Joint offloading and resource allocation for satellite assisted vehicle-to-vehicle communication[J]. IEEE Systems Journal, 2020, 15(3): 3958-3969.

[0178]

[21] De Sanctis M, Cianca E, Araniti G, et al. Satellite communications supporting internet of remote things [J]. IEEE Internet of Things Journal, 2015, 3(1): 113-123.

[0179]

[22] Cao X, Yang B, Shen Y, et al. Edge-assisted multi-layer offloading optimization of LEO satellite-terrestrial integrated networks [J]. IEEE Journal on Selected Areas in Communications, 2022, 41(2): 381-398.

[0180]

[23] Soni G, Sharma M. Performance Evaluation of a Free Space Optical Link-Based Inter Satellite Link (ISL) across Low Earth Orbit (LEO) [C] / / 2022 2nd International Conference on Power Electronics & IoT Applications in Renewable Energy and its Control (PARC). IEEE, 2022:1-5.

[0181]

[24] Zhang H, Liu R, Kaushik A, et al. Satellite edge computing with collaborative computation offloading: An intelligent deep deterministic policy gradient approach [J]. IEEE Internet of Things Journal, 2023, 10(10): 9092-9107.

[0182]

[25] Gao X, Liu R, Kaushik A. Virtual network function placement in satellite edge computing with a potential game approach [J]. IEEE Transactions on Network and Service Management, 2022, 19(2): 1243-1259.

[0183]

[26] Gao X, Liu R, Kaushik A, et al. Dynamic resource allocation for virtual network function placement in satellite edge clouds [J]. IEEE Transactions on Network Science and Engineering, 2022, 9(4): 2252-2265.

[0184]

[27] Li Q, Wang S, Ma X, et al. Service coverage for satellite edge computing [J]. IEEE Internet of Things Journal, 2021, 9(1): 695-705.

[0185]

[28] Cao X, Yang B, Shen Y, et al. Edge-assisted multi-layer offloading optimization of LEO satellite-terrestrial integrated networks [J]. IEEE Journal on Selected Areas in Communications, 2022, 41(2): 381-398.

[0186]

[29] Wu Y C, Dinh T Q, Fu Y, et al. A hybrid DQN and optimization approach for strategy and resource allocation in MEC networks [J]. IEEE Transactions on Wireless Communications, 2021, 20(7): 4282-42.

Claims

1. A method for offloading aircraft tasks based on DDRL in a low-orbit satellite network, characterized by the following steps: include: (1) Build the LEO low-orbit satellite network system into a LEO satellite mobile edge computing network system; (2) Model the utility maximization problem of the LEO satellite network system as a joint decision-making problem of optimizing task offloading and resource allocation; (3) The joint decision-making problem is transformed into a Markov decision process (MDP). Under the framework of dual deep reinforcement learning (DDRL), the dual deep Q network algorithm (DDQN) is used to learn the optimal task offloading decision. On this basis, the temporal difference triple policy gradient algorithm (TD3PG) is used to obtain the optimal resource allocation strategy. In step (1), the LEO satellite constellation is divided into multiple management domains; each management domain is a sub-network system; any management domain consists of a cluster head satellite and S cluster member satellites equipped with mobile edge computing MEC servers, and the cluster member satellites are represented by a set Indicates that the member satellites There are inter-satellite links (ISLs) between the cluster head satellite and member satellites, as well as between some member satellites. Aircraft User Collection Aircraft users The number of aircraft users is represented by U. Assuming that users are located in remote areas, they can only obtain services by accessing the LEO satellite mobile edge computing network. When a user enters a management domain, they automatically connect to the cluster head satellite, which offloads tasks to member satellites for processing based on the type of tasks the user wants to offload. The LEO satellite network system operates in time slot mode, and the set of time slots is represented by Indicates time slot Time is divided into Y time slots of equal length, each time slot length is τ; Assuming that the LEO satellite constellation has a stable topology and connectivity within a certain period of time, at the beginning of each time slot, the aircraft randomly and parallelly generates tasks, which are independent and indivisible; assuming that the time interval between task arrivals follows an exponential distribution with a mean value of μ, in time slot t, the task generated by user u in time slot t is represented by a triplet Indicates that, where l u,t and c u,t Represent the data size and workload of the task respectively, is the maximum tolerable delay of the task; the total number of tasks uploaded by users during the operation of the LEO satellite network system is represented by K; The task offloading and resource allocation process in the LEO satellite network system includes: first, the user accesses and transmits the task to the cluster head satellite for queuing; then, the cluster head satellite offloads the task from the task waiting queue to the member satellites; finally, the task is processed on the member satellites; where: When the cluster head satellite receives the user task, it puts the task to be unloaded into the unloading queue N(t); the decision matrix symbol is used to unload the task from the cluster head satellite to the member satellite. Represents the matrix Indicates whether to offload the task of the i-th user to the j-th member satellite for processing. Indicates that the task of the i-th user is offloaded to the j-th member satellite for processing. Indicates that the task of the i-th user will not be offloaded to the j-th member satellite for processing; When the task is unloaded by the cluster head satellite to the member satellite s, s is the unloaded task Allocate computing resources; set tasks The computing resource allocation strategy is used It means that it is a task The computing resources provided are And there is f s (max) Indicates the maximum computing resources possessed by a member satellite s; When a task is offloaded to a member satellite, the orthogonal computing resources allocated for the task will be retained and occupied for multiple time periods until the task calculation is completed; The computing resources allocated to the member satellites in time slot t cannot exceed the current available resources, which can be expressed as Among them, φ s,t represents the resources occupied by satellite s in time slot t; In the LEO low-orbit satellite network system, a. The communication between the user and the cluster head satellite is: The user uploads the task through the link between the cluster head satellite and the cluster head satellite; the cluster head satellite receives the signal y from user u in time slot t. u,t Expressed as Among them, p u and s u Represent the transmission power and data signal of user u respectively; N0~(0,σ 2 ) is the additive white Gaussian noise (AWGN) of the channel, σ 2 is the noise variance; h u,t represents the AWGN channel between user u and the cluster-head satellite in time slot t, which is expressed as h u,t =d u or u (d u,t ) -β (3) Among them, δ u ~(0,1) is a complex Gaussian variable with Rayleigh fading, η u is the shadow fading that follows the log-normal distribution, β is the path loss exponent; d u,t is the actual distance between user u and the cluster head satellite in time slot t; Assume that the user can only start transmitting requests at an elevation angle α, where α is the minimum angle symmetrical with respect to the line connecting the center of the Earth and the user's position. Let θ represent the central angle corresponding to the arc covered by the satellite, R is the radius of the Earth, H is the satellite orbit height, and α is known. The relationship between θ and α is expressed as Therefore, the actual distance between the cluster head satellite and user u at time slot t is Assuming that the satellite spectrum is orthogonally allocated to all users and the total communication bandwidth of the cluster-head satellite is evenly distributed among the currently associated users, according to the Shannon formula, the data transmission rate from user u to the cluster-head satellite is expressed as: Where B represents the total bandwidth of the user and satellite transmission link frequency band; the worse channel h u,t and a larger number of associated users U will result in a slower data transmission rate; Considering the delay problem, let c represent the speed of light, then the time delay between user u in time slot t and the cluster head satellite is At the same time, the task The transmission energy consumption from user u to the cluster head satellite is b. Communication between satellites: N(t) is used to store the tasks that have been uploaded to the cluster head satellite in time slot t but have not been unloaded. The unloading queue N(t+1) in the next time slot is obtained by subtracting the amount of unloaded task data B(t) and adding the amount of task data newly uploaded to the cluster head satellite. get: in, Represents the relationship between time t and task The associated task arrival rate follows a Poisson distribution, and τ is the length of a time slot; Assuming that the CPU frequency of the cluster head satellite is f′ when processing the unloading queue in time slot t, the amount of task data processed by the cluster head satellite in the unloading queue in one time slot is Where J represents the number of CPU cycles required to process each bit of task data; therefore, the queue delay of the unloading queue N(t) is expressed as Among them, X N(t) Indicates the queue length, It represents the average task arrival rate of queue N(t), which can be expressed as At the same time, energy consumption is not considered during task queuing; Assume that the task is only unloaded once from the cluster head satellite to the member satellite s; after the cluster head satellite makes the unloading decision, the task It will be unloaded from the unloading queue of the cluster head satellite to the member satellite for processing; then the time delay of user u's task from the cluster head satellite to the unloading member satellite in time slot t is expressed as in, represents the dynamic distance between the member satellite s and the cluster head satellite in time slot t; at the same time, let p be the transmission power of the cluster head satellite, then the task The transmission energy consumption is expressed as In time slot t, the computing resources allocated to user u by unloading satellite s are Then process the task The calculation time is Execute tasks on uninstalled satellites The computational energy consumption is expressed as In step (2), Task The total delay from access to unloading is T u,t It is composed of formulas (7), (11), (13) and (15), and is expressed as And there is That is, the end-to-end delay of each task does not exceed the maximum tolerable delay. This condition is used as a sign of whether the task is completed or not, and a binary variable a is used to indicate whether the task is completed or not. u,t It is defined as If the delay constraint is satisfied, the task is considered to be successfully processed. u,t =1, otherwise the task is considered failed. u,t =0; The system energy consumption generated in this process is composed of formulas (8), (14) and (16), which is expressed as Then, the task The end-to-end latency and system energy consumption are determined by the joint strategy of task offloading and resource allocation; At the beginning of each time slot, the cluster head satellite decides to which member satellite to offload the task and how much computing resources to allocate for the task; The system cost includes the sum of the delay and energy consumption of all tasks during the system operation. The joint optimization of task offloading and resource allocation is described as the system cost minimization problem, which is expressed as Among them, ξ∈[0,1] is used to weigh the proportion of energy consumption cost and delay cost in the system cost; The problem of minimizing system cost is replaced by the problem of maximizing system utility P1, which is modeled as Among them, λ is the overhead factor, which is used to solve the problem that the task delay and energy consumption cannot be taken into account due to unfinished tasks. λ is a constant and sufficiently large value. Constraints (21a) and (21b) indicate that each user can offload at most one task to a LEO satellite. (21c) indicates that the resources allocated by the satellite to the task do not exceed the maximum resources owned by the satellite. (21d) indicates that the sum of the computing resources allocated by each satellite to the current offloaded task must not exceed the current available resources of the satellite. Training and optimization of the DDRL framework: The DDRL framework consists of two modules: DDQN and TD3PG. DDQN is used for offloading decision-making problems, while TD3PG is used for computing resource allocation problems. The interaction process between user tasks and system environment includes: Step 1: DDQN takes the environment state as input and iteratively provides services to each user; Step 2: DDQN processes the information and outputs appropriate offloading decisions for each user; Step 3: The environmental state and offloading decision are processed as input by the TD3PG algorithm; Step ④: TD3PG gives appropriate computing resource allocation; Step 5: DDRL outputs the final joint decision; In each interaction process, first, the task offloading action is performed through DDQN Then, perform resource allocation actions according to TD3PG Task From state S t Action A u,t Enter state S t+1 The reward value r(S t ,A t ,S t+1 ) Use the symbol r u,t When DDQN interacts with the environment, it follows the ε-greedy strategy to perform the action of the offloading decision; ε is a probabilistic decision factor ε, which is between 0 and 1; In the DDQN learning process, ε is used to represent the probability of randomly selecting an action, and 1-ε is used to represent the probability of executing the action given by the DDQN main network. When exploring the system environment for the first time, set a larger value for ε. As learning accumulates, the value of ε gradually decreases, and the agent will make more use of the knowledge it has learned to choose the action with the greatest reward. During the TD3PG learning process, since TD3PG is suitable for continuous actions Then it adds random noise when exploring the system environment. The noise obeys the normal distribution with variance σ, and the value of σ gradually decreases during the learning process. After the interaction is completed, an observation value for DDQN and an observation value for TD3PG are generated, which are stored in independent replay buffers B1 and B2 respectively; When task offloading and resource allocation are based on DDRL, First, update the DDQN: Set up the Q network and target network DDQN collects quads from replay buffer B1 For Q and Learning; when using samples to update Q, DDQN Calculated value y u,t As the target value, it is expressed as Where γ is the decay factor; the loss function for updating Q is the square error function, defined as Next, calculate the gradient and update the Q network every g1 steps. At the same time, by setting Update every G1 step Then, update TD3PG: TD3PG consists of two independent value networks C1 and C2, a strategy network A and its corresponding target network as well as Actor Network A predicts resource allocation strategy Critc networks C1 and C2 determine the q-value of the strategy generated by the Actor network during the learning process; Before TD3PG training, some values ​​are sampled from the playback buffer B2; during training, some values ​​are sampled from the target network and Calculate the target value in Then, the target value y u,t Compared with the q value, the difference between the two is the time difference TD error δ; this function is a square error, defined as The gradient of the above formula is used to update the Critic network C1 and C2; To update the Actor network A, calculate the gradient of the value q1=C1(s,a) predicted by the Critic network C1; use the gradient ascent method to minimize -q1 to maximize the value of q1; train the networks A, C1 and C2 and update them every g2 steps; at the same time, the target network and The value network is updated once every G2 steps; the policy network is updated once every μ updates.

2. The method for offloading aircraft tasks based on DDRL in a low-orbit satellite network according to claim 1, characterized in that In step (3), the problem transformation method is: Due to the correlation of decisions in adjacent time slots, the task offloading and resource allocation process is transformed into an MDP problem; MDP consists of five elements: state space S, action space A, state transition probability P, reward function R, and discount factor γ; at time slot t, First, the cluster head satellite obtains the status information of the member satellites, including load conditions and remaining resources, as well as uploaded user mission information; Then, the cluster head satellite outputs the task offloading and resource allocation decision, and returns the resource allocation decision to the member satellites while making the offloading action. The member satellites allocate resources for task processing according to the allocation decision. Finally, the member satellites return the results to the users; The cluster head satellite learns and implements strategies through continuous interaction with the system environment and obtains rewards to improve the strategy.

3. The method for offloading aircraft tasks based on DDRL in a low-orbit satellite network according to claim 2, characterized in that The key elements of MDP are described as follows: State space: At time slot t, the cluster head satellite has the state information of all aircraft users and member satellites from a holistic perspective; the system state is expressed as L u,t represents the geographical location of user u at time slot t. The location information is described as a two-dimensional vector containing the longitude and latitude in the geodetic coordinate system, with the elevation set to 0; D u,t is a U-dimensional vector, each element of which is the distance between user u and the cluster head satellite at time slot t; Represents the attribute information of the task uploaded by user u in time slot t; is an S-dimensional vector, which refers to the distance between satellite s and the cluster head satellite at time slot t; s,t is an S-dimensional vector, indicating the number of resources currently occupied by each member satellite; N(t) represents the queue status of the current unloading queue; Action space: The action refers to the selection of unloading satellites and resource allocation plan made by the cluster head satellite in time slot t, which is defined as in and They represent task offloading and resource allocation actions respectively. In the process of dynamic task offloading and resource allocation, the cluster head satellite needs to observe the system environment S t Determine the satellite to be unloaded and make the decision based on the integrated system environment observation information S t , determine to unload the satellite as a task allocated computing resources; Reward: In system state S t Assign user tasks Take action A u,t After that, the system environment enters a new state S t+1 And return the corresponding reward; put the system in state S t The total cost under ψ(S t ) means that when the system undergoes action A u,t Enter state S t+1 After that, state S t and S t+1 There is a cost difference ψ(S t )-ψ(S t+1 ); then, the design of the reward function is based on the cost difference of the system state transition process, expressed as r(S t ,A t ,S t+1 )=ψ(S t )-ψ(S t+1 ) When r(S t ,A t ,S t+1 )>0, when the system cost decreases with time slots, the action gets a positive reward value, otherwise the reward value is negative, indicating that the action is punished.

4. The method for offloading aircraft tasks based on DDRL in a low-orbit satellite network according to claim 1, characterized in that In the management domain of the LEO satellite network system, cluster head satellites are regularly elected; In the LEO satellite network system, if a user is covered by multiple management domains, the cluster head satellite of the management domain with high channel quality is selected by default for access.

Images