Unmanned aerial vehicle trajectory optimization and task offloading method facing air-space-ground integrated network

Abstract

The invention belongs to the technical field of air-space-ground integrated networks, and discloses an unmanned aerial vehicle track optimization and task offloading method for an air-space-ground integrated network. According to the method, the task unloading efficiency in the air-space-ground integrated network is optimized, optimization of energy consumption and time delay is focused on, various conditions of the air-space-ground integrated network are comprehensively considered, a decoupling communication mode is considered, and the influence of service interruption is reduced. And constructing an optimization model to minimize the weighted sum of time delay and energy consumption. In order to improve the collaboration of heterogeneous agents in a space-air-ground integrated network, based on a hierarchical reinforcement learning framework, a hierarchical learning-based two-stage Qmix algorithm is proposed, and the collaboration of a low-orbit satellite and an unmanned aerial vehicle is reinforced. According to Pycharm platform simulation and multi-scene verification, the method has remarkable advantages in average response time delay, energy consumption and QoS indexes, the network performance is effectively improved, and user requirements are met.

Description

Technical Field

[0001] The present invention relates to the field of air-ground-integrated network technology, and in particular to a method for optimizing the trajectory of unmanned aerial vehicles (UAVs) and offloading tasks in an air-ground-integrated network. Background Art

[0002] With the rapid development of Internet technology, the amount of task data of intelligent terminal devices is increasing. Due to reasons such as device hardware technology, terminal devices cannot provide sufficient computing resources for emerging computing-intensive tasks and delay-sensitive tasks. Since 2008, cloud computing has been applied to this field. Cloud computing centers rely on powerful computing power to break through the resource limitations of intelligent devices and provide diversified and efficient applications and computing services. However, although traditional cloud computing can provide sufficient resources for end users, it cannot meet the latency requirements of applications for rapid response. Therefore, Mobile Edge Computing (MEC) has emerged as a new technology. MEC is a technology that moves computing tasks from the cloud center to the edge device for computing, realizing the close integration of computing platforms and devices, providing powerful computing and storage capabilities for terminal devices while reducing computing energy consumption and latency, so that users can obtain higher quality of service (QoS).

[0003] Traditional MEC technology is difficult to deploy in complex terrains such as remote mountainous areas and oceans, and cannot provide Internet services to end users. In addition, network infrastructure is also vulnerable to natural disasters such as earthquakes and floods, resulting in service interruptions. Therefore, a new technology is urgently needed to break through this limitation. With the vigorous development of low-orbit satellite-related technologies by major technology companies in recent years, as of May 2024, the number of satellites in the world has exceeded 9,000, of which low-orbit satellites account for more than 90%. The Space-Air-Ground Integrated Network (SAGIN) has become a hot research object for the next generation of mobile communication networks. Integrating a variety of mobile communication devices with little environmental restrictions, SAGIN is expected to break through the above technical difficulties and alleviate the limitations of existing network communication technologies. The SAGIN architecture mainly consists of three parts: a space-based network composed of low-orbit satellites, an air-based network composed of unmanned aerial vehicles (UAVs), and a ground-based network composed of ground users and traditional ground communication equipment. Among these three parts, SAGIN is based on the ground-based network, combined with the wide coverage of satellite-based networks and the advantages of flexible deployment of UAVs, and achieves seamless coverage of the air, space and ground through the deep integration of multi-level networks. Therefore, the integration of MEC technology and SAGIN can provide services for terminal devices by taking advantage of SAGIN's flexible deployment, low cost, wide range and multi-layer network.

[0004] At present, many valuable research results have been achieved in the application of MEC technology in SAGIN. However, there are still a series of key issues that need to be resolved: First, during data transmission, due to the relative movement of network nodes, the communication link between nodes will be disconnected, which will cause service interruption problems in the network, thereby greatly reducing the service quality of users. At present, relevant research lacks research on data transmission after the link is disconnected, resulting in higher latency. Secondly, due to the complexity and mobility of the SAGIN network structure, it is difficult for existing algorithms to quickly solve an efficient task offloading solution, and in this process, the coordination of multiple heterogeneous devices in SAGIN is often ignored, resulting in a decrease in the overall system performance. Solving these problems requires in-depth research and innovation to improve the performance and efficiency of the system to meet the growing needs of mobile applications. Summary of the invention

[0005] To solve the above problems in SAGIN, the present invention proposes a UAV trajectory optimization and task offloading method for the integrated air-ground-space network. The core is to use the nodes of each layer of SAGIN to provide efficient task offloading services for ground users. UAV is responsible for collecting user data and deciding whether to calculate in its own calculation or upload it to LEO. The two return the calculated results to the ground user to reduce the delay and energy consumption of task processing.

[0006] The technical solution of the present invention is as follows: a method for optimizing the trajectory of unmanned aerial vehicles and offloading tasks for an air-ground integrated network, establishing a network model, a node communication model, a task offloading energy consumption model and a delay model;

[0007] The optimization problem is to minimize the energy consumption and average delay of completing the task. The timing prediction algorithm and multi-agent deep reinforcement learning algorithm are used to solve the optimization problem and obtain the optimal trajectory optimization and task offloading strategy.

[0008] The network model is divided into three layers: ground layer, air layer and space layer;

[0009] The ground layer includes ground users;

[0010] The aerial layer is mainly composed of UAVs. UAVs establish communication links with ground users, receive tasks uploaded by ground users, and calculate the tasks themselves or pass them to upper-layer satellites for calculation based on decisions. At the same time, there are communication links between UAVs and satellites to achieve information transmission and interaction.

[0011] The space layer includes multiple LEO satellites in the sky; the iTransformer-based time series prediction model is deployed in the LEO satellites; the LEO satellites are regarded as high-level intelligent entities, which make high-level decisions at regular intervals based on the observed state information and the user distribution predicted by the iTransformer-based time series prediction model to guide the low-level strategies of the drones;

[0012] The network model includes N ground users, M drones and L low-orbit satellites. The ground user set is represented as Its three-dimensional coordinates are expressed as l n (t) = (x n (t),y n (t),h n (t)), the coordinates of the ground users are randomly assigned at the beginning of each round, and then move randomly at the beginning of each time slot. The set of aerial drones is represented as Its three-dimensional coordinates are expressed as l m (t) = (x m (t),y m (t),h m(t)) is represented by moving at a speed V in each time slot and is responsible for collecting data from ground users. The set of low-orbit satellites above is represented by l∈L={1,2,...,L}, and its three-dimensional coordinates are represented by l l (t) = (x l (t),y l (t),h l (t)), LEO moves in a fixed orbit; ground users can establish communication with LEO satellites only when they enter the communication window.

[0013] In the node communication model, the communication link between the UAV and the ground user is a line-of-sight link LoS or a non-line-of-sight link NLoS; the path loss of Los is The path loss corresponding to NLoS is where β LoS and β NLoS are two different attenuation factors that distinguish LoS and NloS links, α mn (t) is the power gain, and the average path loss for the communication link is defined as:

[0014]

[0015] Among them, P LoS (t) is the probability of establishing a LoS channel between the UAV and the user in a specific network environment, and the probability of establishing a NLoS channel between the two is P NLoS (t) = 1-P LoS (t);

[0016] Decouple the communication link between the UAV and the ground user, separate the uplink and downlink, so that each ground user can establish uplink and downlink communication with different UAVs, and define a binary variable To indicate whether UAVm is associated with user n, it is expressed as:

[0017]

[0018] When (·) = D, it indicates that the communication link is a downlink, and when (·) = U, it indicates that the link is an uplink. In each time slot, each user is only allowed to establish an uplink or downlink with one UAV. At the same time, due to the limitations of UAV hardware factors, the UAV can only establish connections with Q ground users at most in each time slot.

[0019] Considering the channel interference during decoupling, the communication links between drones and between drones and satellites are established in line-of-sight transmission mode. Therefore, the information transmission rate between drones and between drones and satellites is defined as follows:

[0020]

[0021] Among them B back is the backhaul link bandwidth, σ 2 is the noise power, p o (t) is the transmission power;

[0022] The information transmission rate between the UAV and the ground user is expressed as:

[0023]

[0024] is the link connection status between drone m and user n at time t, B is the channel bandwidth, It is the signal to noise plus interference ratio. For uplink and downlink, the corresponding signal to noise ratio is:

[0025]

[0026] Among them, p n (t) represents the transmission power of user n, p m (t) represents the transmission power of UAV m, represents the interference caused by other users’ uplink transmission to UAV m, Indicates the interference caused by other drones’ downlink transmission to drone m, represents the self-interference of drone m, represents the interference caused by the downlink transmission of other drones to user n, represents the interference caused by other users' uplink transmission to user n, represents the self-interference of user n;

[0027] When the task offloading energy consumption model is initialized, each ground user is randomly assigned a computing task with a certain amount of data. The ground user needs to upload the task to the UAV via the uplink, and then the UAV decides whether to upload the task to the LEO satellite for calculation or to calculate it itself. During the entire task calculation cycle, the energy consumed by the ground user and the UAV is mainly considered, which is specifically divided into four aspects:

[0028] (1) Energy consumption of ground user data transmission: The user needs to transmit all his / her data to the UAV, and the data volume is I n , the power of user uploading data is defined as p n (t), so the transmission energy consumption is defined as:

[0029]

[0030] in Indicates the information transmission rate of the uplink;

[0031] (2) Energy consumption of UAV data transmission: When the UAVs that establish uplink and downlink links with the same user are different, the UAV in the uplink transmits data to the corresponding UAV in the downlink, or forwards data to the LEO satellite, and its transmission power is defined as p m (t), the energy consumption is:

[0032]

[0033] The energy consumption of the UAV to perform calculations and return the results to the ground user when the task calculation is completed is:

[0034]

[0035] I m Indicates the amount of data that drone m needs to transmit, Indicates the downlink information transmission rate, Indicates the information transmission rate between drones or between drones and satellites;

[0036] (3) Energy consumption of drone computing tasks: Assume that the amount of data required for the drone computing task is TA m , the calculation rate is Calculate the power as P c , the energy consumption of the computing task is defined as:

[0037]

[0038] (4) UAV flight energy consumption: The flight energy consumption of UAV is greater than that of hovering energy consumption, so the flight energy consumption of UAV is mainly considered. The flight power of UAV is defined as P f , the flight energy consumption is defined as:

[0039]

[0040] The delay model mainly includes two parts: task calculation delay and information transmission delay. The task calculation delay involves the UAV calculation delay and the satellite calculation delay, which are defined as: The information transmission delay is divided into uplink delay, downlink delay, and transmission delay between drones and between drones and satellites, which are defined as:

[0041]

[0042] Among them, I n represents the size of the task uploaded by the ground user, I m Indicates the amount of data transmitted by the drone, I' m Indicates the amount of data transmitted between drones. Indicates the amount of data transmitted by the drone to the low-orbit satellite;

[0043] Therefore, the cycle delay for each task to complete is defined as:

[0044]

[0045] The time series prediction algorithm divides the ground area into multiple small blocks, takes each small block as a unit, uses a real data set to simulate the number of ground users in the unit, and uses it as historical time series data to predict through the iTransformer time series prediction model; the iTransformer-based time series prediction model is a Transformer-based time series prediction architecture, which does not change the Transformer network architecture, but converts the role of the attention mechanism and the feedforward network; different variables are considered separately, each variable is encoded into an independent token, the attention mechanism is used to model the correlation between different variables, and the feedforward network is used to model the time series correlation of variables to obtain a sequence time series representation; the historical time series data is input, and the number of users in each unit at the future moment is output.

[0046] The optimization problem is specifically:

[0047] By optimizing the flight trajectory of the UAV and the offloading decision variables, the computational offloading constraint optimization problem is obtained with the goal of minimizing the total energy consumption of the system task and the weighted sum of the average delay, while ensuring the communication resource constraints, computing resource constraints, LEO satellite coverage constraints, and the maximum tolerable delay constraints of the task; the optimization problem is as follows:

[0048]

[0049] The constraints are:

[0050]

[0051] Among them, constraint C1 means that the power of UAV at any moment cannot exceed its maximum power; constraint C2 means the UAV battery capacity limit, constraint C3 means the UAV maximum coverage limit, constraint C4 means the task calculation tolerance delay, constraint C5 means each user can only establish an uplink or downlink connection with one UAV, constraint C6 means the UAV flight speed limit, and constraint C7 means the instantaneous power consumption limit of the user and UAV;

[0052] The determined optimization problem is modeled as a partially observable Markov decision process POMDP, and the MDP is simplified to transform it into a model-free process; the state space, action space and reward function are defined as follows;

[0053] 1) State space: The state space includes all network factors that affect the agent's decision-making, expressed as Including the location coordinates of UAV, LEO, and ground users, the size of the mission, and energy consumption;

[0054] 2) Action space: The action of UAV flying and the action of establishing connection with ground users are respectively represented as

[0055] 3) Reward function: Aims to minimize the objective function, which is used as the reward function and is defined as:

[0056] The multi-agent deep reinforcement learning algorithm is a two-level Qmix algorithm based on hierarchical learning; it includes two parts: a high-level decision network and a low-level decision network;

[0057] Consider low-orbit satellites as high-level agents in a high-level decision-making network, and drones as low-level agents in a low-level decision-making network;

[0058] The high-level decision network and the low-level decision network respectively include an agent's decision network RNN ​​and a mixing network for fitting multi-agent Q values;

[0059] The RNN network loop inputs the current observation o and the action a of the agent at the previous moment t-1 , output Q value; the mixing network inputs the Q values ​​of multiple agents and the state information of the global environment, and outputs the global Q value Q tot ;

[0060] An encoder based on a multi-head attention mechanism is added to the mixing network to reduce the dimension of the state information of the global environment, and the multi-head attention mechanism is used to extract effective spatiotemporal feature information. At the same time, the results predicted by the time series prediction algorithm are used as part of the reward function of the high-level decision network to guide the agent to identify areas with high returns in the early stage of strategy exploration and avoid the impact of sparse environmental rewards. The loss function of the mixing network is defined as:

[0061]

[0062] The mixing network adopts the network update method of DQN, θ represents the current network parameters, and b represents the number of samples; R is the reward function, θ - Represents the target network; the high-level decision network and the low-level decision network jointly learn the same task and share a target network to update network parameters;

[0063] The high-level decision network is deployed on the low-orbit satellite, and the low-level decision network is deployed in the drone. The low-orbit satellite, as a high-level intelligent agent, makes a high-level decision every K time interval based on the observed state information and the user distribution predicted by the iTransformer-based time series prediction model to guide the drone's low-level strategy for the next k steps. The drone takes its own actions at each moment based on its own observation information and the reward function given by the high-level decision network in the low-orbit satellite. The reward value obtained by the high-level decision network is evenly distributed to the low-level decision network in the future k time slots. The reward function of the low-level decision network is expressed as:

[0064]

[0065] Where R H The reward function representing the high-level decision network is defined as:

[0066]

[0067] in, Indicates the number of users covered by the drone, It represents the difference between the predicted result of the time series prediction model based on iTransformer and the real data, and j and k represent their weights.

[0068] Beneficial effects of the present invention: In summary, the present invention provides a comprehensive and accurate method to deal with the UAV flight trajectory and task offloading decision-making problems in the air-ground integrated network. The present invention optimizes the flight trajectory and offloading strategy of the UAV, and continuously learns and adapts to the environment, so that the UAV can dynamically adjust its flight trajectory and offloading strategy according to factors such as the distribution of ground users, changes in task requirements, and the state of the network. This method can not only meet the computing needs of users in a specific area, but also make the offloading strategies of UAVs and satellites highly synergistic through the designed algorithm, significantly improving the performance of the overall network and better meeting the user's needs for high energy efficiency and low latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A structural diagram showing the overall network model;

[0070] Figure 2 Display the network structure based on iTransformer time series prediction model;

[0071] Figure 3 Show a two-level Qmix algorithm design method based on hierarchical learning;

[0072] Figure 4 Shows a two-level Qmix algorithm network structure based on hierarchical learning;

[0073] Figure 5 Displays the change of algorithm reward value under specified parameters, where HA_Qmix is ​​the method of the present invention;

[0074] Figure 6 Display the energy consumption of multiple ground users;

[0075] Figure 7 Displays the delay of multiple ground users. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and implementation cases.

[0077] The decoupled communication mode between drones and ground users is considered in the invention scenario. When unloading tasks, ground users can select different drones to establish uplink and downlink connections. In a task calculation cycle, if the link is disconnected during uplink transmission, the drone needs to forward the data to the new uplink drone, and the ground user continues to transmit the data that has not been successfully transmitted. After the uplink drone completes the task, it needs to transmit the calculation results to the downlink drone, which will be delivered to the user by the downlink drone. This can effectively break through the limitations of full-duplex coupled communication and reduce the problem of long waiting time for users due to link disconnection. At the same time, the research problem is modeled as an optimization problem, the goal is to minimize the average delay and weighted sum of energy consumption for users to complete the calculation task, and establish uplink and downlink links between drones and ground users as two actions in the action space of the intelligent agent, and jointly consider the relationship between drones and users and flight trajectories to ensure the continuity of task offloading services and high-quality data delivery.

[0078] In order to solve the problems related to task offloading under SAGIN, the present invention proposes a two-level Qmix algorithm based on hierarchical learning based on the value decomposition framework HAVEN of the multi-agent complete cooperation problem of hierarchical reinforcement learning, which includes two parts: a high-level decision network and a low-level decision network. As a high-level agent, the low-orbit satellite makes a high-level decision every K time according to the observed state information and the user distribution predicted by the prediction module, guiding the low-level strategy of the next k steps of the drone. The drone obtains its own action at each moment according to its own observation information and the reward function given by the high-level strategy. At the same time, a time series prediction model based on iTransformer is used to predict the user distribution and guide the algorithm training. By designing the reward function in the two-layer hierarchical structure, a dual coordination mechanism of inter-level (low-orbit satellite and drone) and inter-subject (between drones) strategies is introduced, which can ensure that the agent takes into account the actions and goals of other agents when performing tasks, thereby improving the coordination of the agents.

[0079] Step 1: Build a system model

[0080] Step 1.1: Network Model

[0081] The scenario considered by the present invention is: ground users have a certain number of computationally intensive tasks or delay-sensitive tasks that are difficult to complete alone. The integrated air-ground network is used to provide computing services for ground users. Relying on the good maneuverability of drones and the high coverage of low-orbit satellites, continuous and high-quality services are provided to ground users to meet the delay and energy consumption needs of ground users' computing tasks. After the ground user uploads the task to the corresponding drone, the drone, as the decision maker of the offloading decision, decides whether the task is calculated in the drone itself or delivered to the upper low-orbit satellite for calculation. The overall SAGIN is divided into three layers:

[0082] Ground layer: includes ground users who have computationally intensive or delay-sensitive tasks that are difficult to complete by themselves and need to obtain computing services with the help of the integrated air-ground-space network. For example, equipment in a smart factory on the ground may need to process a large amount of production data in real time, but its own computing resources are limited, so it can upload the task to other nodes in the network for processing.

[0083] Aerial layer: mainly composed of unmanned aerial vehicles (UAVs). UAVs have good mobility and play a key role in the task offloading process. They can establish communication links with ground users, receive tasks uploaded by ground users, and calculate the tasks themselves or pass them to upper-layer satellites for calculation based on decisions. At the same time, there are communication links between UAVs and between UAVs and satellites to achieve information transmission and interaction.

[0084] Space layer: includes multiple low-orbit satellites (LEO) in the sky. Low-orbit satellites have the characteristics of high coverage, can observe the regional status of a large range, and have relatively strong computing power. In the entire network system, low-orbit satellites are regarded as high-level intelligent entities. They make high-level decisions at regular intervals based on the observed status information and the user distribution predicted by the prediction module, guide the low-level strategies of drones, and play an important role in optimizing drone flight trajectories and task offloading strategies to ensure continuous and high-quality services for ground users and meet the latency and energy consumption requirements of ground users' computing tasks.

[0085] Different computing nodes have different computing and communication capabilities, and more efficient task processing is achieved through collaboration. The network model considered in this paper includes N ground users, M drones, and L low-orbit satellites. The ground user set is represented by Its three-dimensional coordinates are expressed as l n (t) = (x n (t),y n (t),h n(t)), the coordinates of the ground users are randomly assigned at the beginning of each time slot, and then move according to the random model. The set of aerial drones is represented as Its three-dimensional coordinates are expressed as l m (t) = (x m (t),y m (t),h m (t)) is represented by moving at a speed V in each time slot and is responsible for collecting data from ground users. The set of low-orbit satellites above is represented by l∈L={1,2,...,L}, and its three-dimensional coordinates are represented by l l (t) = (x l (t),y l (t),h l (t)), LEO moves in a fixed orbit. Due to the fast movement speed and limited coverage of LEO satellites, ground users are not always connected to LEO satellites. Only when LEO satellites and ground users maintain a special geometric relationship, that is, when LEO satellites enter the communication window, ground users can establish communication with them.

[0086] Step 1.2: Communication Model

[0087] The communication link between the UAV and the ground user is generally considered to be a line-of-sight link (LoS) or a non-line-of-sight link (NLoS), which is closely related to the network environment. The path loss of Los is The path loss corresponding to NLoS is where β LoS and β NLoS are two different attenuation factors that distinguish LoS and NloS links, α mn (t) is the power gain, so the average path loss for this communication link is defined as:

[0088]

[0089] Where P LoS (t) is the probability that the UAV and the user establish a LoS channel in a specific network environment. Correspondingly, the probability that the two establish a NLoS channel is P NLoS (t) = 1-P LoS (t).

[0090] Limited by the impact of full-duplex coupling channels on task offloading scenarios, the present invention intends to decouple the communication link between the UAV and the ground user, separate their uplink and downlink, so that each ground user can establish uplink and downlink communication with different UAVs, and define a binary variable To indicate whether UAVm is associated with user n, it can be expressed as:

[0091]

[0092] When (·) = D, it indicates that the communication link is a downlink, and when (·) = U, it indicates that the link is an uplink. It is worth noting that in each time slot, each user is only allowed to establish an uplink or downlink with one UAV. At the same time, due to the limitations of UAV hardware factors, the UAV can only establish connections with a maximum of Q users in each time slot.

[0093] Different from the conventional full-duplex coupled communication mode, the decoupled communication mode considered in the present invention may have some additional channel interference.

[0094] (1) Uplink: The interference comes from the interference of other uplinks, the interference of other downlinks of the drone, and the self-interference of the channel, which can be expressed as:

[0095]

[0096] (2) Downlink: Similar to the uplink, the relevant interference in the downlink experienced by the user can be expressed as:

[0097]

[0098] Therefore, the information transmission rate between the drone and the user can be expressed as:

[0099]

[0100] in It is the signal to noise plus interference ratio. For uplink and downlink, the corresponding signal to noise ratio is:

[0101]

[0102]

[0103] Since the communication between drones is usually in a free and open space, the communication links between drones and between drones and satellites will be established in line-of-sight transmission mode, so the information transmission rate is defined as follows:

[0104]

[0105] Step 1.3: Task offloading model

[0106] The models for task offloading mainly include task offloading energy consumption model and delay model.

[0107] Step 1.3.1: Energy consumption model

[0108] During initialization, each ground user will be randomly assigned a computing task with a certain amount of data, represented by In ,The ground user needs to upload the task to the UAV through the uplink, and then the UAV decides whether to upload the task to the LEO satellite for calculation or to calculate it itself.,During the entire task calculation cycle, the present invention mainly considers the energy consumed by the ground user and the UAV, which can be specifically divided into four aspects:

[0109] (1) Energy consumption of ground user data transmission: The user needs to transmit all his / her data to the UAV, and the data volume is I n , the power of user uploading data is defined as p n (t), so the transmission energy consumption can be defined as:

[0110]

[0111] (2) Energy consumption of UAV data transmission: When the UAVs that establish uplink and downlink links with the same user are different, the uplink UAV needs to transmit data to the corresponding downlink UAV, or forward data to the LEO satellite, and its transmission power is defined as p m (t), the energy consumption is:

[0112]

[0113] If the UAV performs the calculation, the energy consumption required to return the result to the ground user when the task calculation is completed is:

[0114]

[0115] (3) Energy consumption of drone computing tasks: Assume that the amount of data required for the drone computing task is TA m , the calculation rate is Calculate the power as P c , the energy consumption of the computing task can be defined as:

[0116]

[0117] (4) UAV flight energy consumption: The flight energy consumption of UAV is greater than that of hovering. Therefore, the present invention mainly considers the flight energy consumption of UAV. The flight power of UAV is defined as P f , the flight energy consumption can be defined as:

[0118]

[0119] Step 1.3.2: Delay Model

[0120] The delay part mainly includes two parts: task calculation delay and information transmission delay. The task calculation delay is designed into the UAV calculation and satellite calculation delay, which are defined as: The information transmission delay is divided into uplink delay, downlink delay, and transmission delay between drones and between drones and satellites, which are defined as:

[0121]

[0122] Therefore, the cycle delay for each task to complete is defined as:

[0123]

[0124] Step 2: Problem Definition

[0125] For each mathematical model, by optimizing the flight trajectory of the UAV and the unloading decision variables, the computational offloading constraint optimization problem with the goal of minimizing the total energy consumption of the system task and the weighted sum of the average delay is obtained, while ensuring the communication resource constraints, computing resource constraints, LEO satellite coverage constraints, and the maximum tolerable delay constraints of the task. The optimization problem is given as:

[0126]

[0127] The constraints are:

[0128]

[0129]

[0130] Among them, constraint C1 means that the power of UAV at any moment cannot exceed its maximum power; constraint C2 means the battery capacity limit of UAV, constraint C3 means the maximum coverage range limit of UAV, constraint C4 means the task calculation tolerance delay, constraint C5 means each user can only establish an uplink or downlink connection with one UAV, constraint C6 means the flight speed limit of UAV, and constraint C7 means the instantaneous power consumption limit of user and UAV.

[0131] Step 3: Time series prediction model based on iTransformer

[0132] The ground area is divided into multiple small blocks, and the number of users is simulated using real data sets, which are used as time series data and predicted using the iTransformer-based time series prediction model. The result is used as one of the reward function parameters of the subsequent algorithm to accelerate the convergence of the subsequent algorithm.

[0133] The iTransformer-based time series prediction model is a brand-new Transformer-based time series prediction architecture. It does not change the Transformer network architecture, but transforms the role of the attention mechanism and the feedforward network. iTransformer considers different variables separately, and each variable is encoded into an independent token. It uses the attention mechanism to model the correlation between different variables, and uses the feedforward network to model the temporal correlation of variable variables to obtain a better sequence timing representation. The iTransformer-based time series prediction model includes an embedding module, a multi-head attention mechanism, a normalization layer, a feedforward neural network, and a mapping module;

[0134] (1) For historical window data Target is pre-data

[0135] (2) First, obtain the sequence encoding of the univariate: Use a simple MLP layer to convert Mapping

[0136] (3) Analyze different variables using multi-head attention mechanism The correlation between Figure 2 As shown, Q, K, and V are all derived from the variable H :n Starting from, the multivariate correlation map is the calculated attention coefficient, which indicates the magnitude of the correlation between different variables. Through this module, the correlation between different variables is integrated to update the embedding vector embedding of the variable.

[0137] (4) Normalization layer: In order to ensure the independence of variables and avoid the influence of sampling misalignment or time lag on the cross-correlation of modeling variables, each variable is standardized separately.

[0138]

[0139] (5) Feedforward neural network: Finally, the feedforward neural network is used to further model the timing effects within each variable to obtain efficient timing representation.

[0140] The process of the time series prediction model algorithm based on iTransformer is as follows:

[0141]

[0142] Step 4: Design of two-level Qmix algorithm based on hierarchical learning

[0143] The determined optimization problem is modeled as a partially observable Markov decision process (POMDP). Most of the current MARL algorithms for POMDP follow the centralized training distributed execution (CTDE) framework, where each agent can utilize all available information during training but can only make decisions based on local observations.

[0144] Typically, POMDP is described as a 4-tuple, including state space, state transition probability, action space, and reward function. In this context, since the state of the agent is high-dimensional and continuous and it is difficult for us to accurately obtain the state transition probability, the MDP is simplified and converted into a model-free process. In order to apply deep reinforcement learning (DRL) to trajectory optimization and task offloading problems, the present invention models the problem as an MDP. In this model, the state space, action space, and reward function are defined as follows.

[0145] State space: The state space includes all network factors that affect the agent's decision, expressed as Including the location coordinates of UAV, LEO, and ground users, the size of the mission, and energy consumption.

[0146] Action space: The action space mainly consists of two parts, namely the UAV flight action and the action of establishing a connection with the ground user. Therefore, the action space at time slot t can be expressed as

[0147] Reward function: Aims to minimize the objective function, so it is used as a reward function and is defined as:

[0148] As the complexity of problems and scenarios increases, multiple heterogeneous agents frequently cooperate to complete the same task. Existing research algorithms lack cooperative relationships between agent levels and only consider cooperative relationships within the levels. Inspired by the human nervous system, when completing a complex task, it is inseparable from the close cooperation of specific brain areas and lower-level organs. The brain, as a high-level strategy, needs to guide the respective low-level strategies. Therefore, when the human body is regarded as a multi-agent system, coordination between and within layers is crucial to solving the task of full cooperation.

[0149] Based on the above content, in the SAGIN scenario, low-orbit satellites are regarded as high-level intelligent agents due to their large coverage, ability to observe the global state, and strong computing power. UAVs are regarded as low-level intelligent agents because of their limited observation range and relatively weak computing power, but the specific strategy implementation needs to be completed by them. Based on a value decomposition framework HAVEN for the multi-agent complete cooperation problem of hierarchical reinforcement learning, a two-level hierarchical Qmix algorithm is proposed, which consists of a high-level decision network and a low-level decision network.

[0150] The network structure of the high-level decision network and the low-level decision network is similar to the traditional value decomposition algorithm Qmix. The Actor network of the intelligent agent adopts the RNN network, which takes the observation at the current moment and the hidden state at the previous moment as the current input to extract the hidden information of the time series. The mix network is equivalent to the Critic network, which receives the Q value of the Agent RNN Network and the current global state st at the same time, and outputs the behavior utility value Q of the joint behavior u of all agents in the current state tot .

[0151] Since the iTransformer-based time series prediction model is designed to predict the user distribution at future moments, the intelligent agent can identify areas with high rewards at the early stage of strategy exploration, avoiding the impact of sparse environmental rewards. The high-level decision network and the iTransformer-based time series prediction model are deployed on low-orbit satellites, and the bottom-level decision network is deployed in drones. The low-orbit satellite, as a high-level intelligent agent, makes a high-level decision every K time based on the observed state information and the user distribution predicted by the prediction module to guide the drone's next k-step low-level strategy. The drone obtains its own actions at each moment based on its own observation information and the reward function given by the high-level strategy. By designing the reward function in the two-layer hierarchical structure, a dual coordination mechanism of inter-level (satellite and drone) and inter-subject (between drones) strategies is introduced, which can ensure that the intelligent agent takes into account the actions and goals of other intelligent agents when performing tasks, thereby improving the coordination of intelligent agents.

[0152] At the same time, when fitting the global q value in the mix network, the global state is pre-encoded by the Transformer encoder, which can extract the features containing the spatial correlation between the state information of different regions at the current moment, and at the same time reduce its dimension, making the algorithm converge faster. The overall training process of the algorithm is as follows:

[0153]

[0154]

[0155] Step 5: Performance Verification

[0156] The simulation implementation of this method is based on the Pycharm platform, and python is used to simulate the scene model. The experiment randomly deploys 5 UAVs, 20 ground users and 3 LEOs in a 1000m*1000m square area, and assumes that the UAV hovers at a fixed altitude of 50 meters. The mission data size of the ground user is 8MB, and the maximum tolerable delay constraint of the ground user is 1000ms. The ground altitude from the low-orbit satellite is 784km, the channel power gain at a reference distance of 1 meter is -80dB, the path loss is 8, the available channel bandwidth of UAV is 300MHz, the available channel bandwidth of LEO is 40GHz, the background noise of ground user communication is 40GHz, the computing power of UAV is assumed to be 4G / s, the computing power of LEO is assumed to be 10G / s, the transmission power of UAV is 1w, and the transmission power of LEO is 3w. At the same time, a comparison algorithm is set in the simulation verification of this method. In order to evaluate the three performance indicators of average response delay, average response energy consumption, and quality of service (QoS), DQN, MADDPG, and Qmix were compared on the Pycharm platform. Figure 5 As shown in the figure, the reward value of this method is better than that of MADDPG and Qmix algorithms during training, and DQN performs the worst; Figure 6 It means that under the above parameter conditions, 10 users are randomly selected from ground users to analyze the task calculation energy consumption of different algorithms. It is found that the task energy consumption of this method is the lowest, Qmix and MADDPG perform relatively poorly, and DQN has the highest energy consumption; Figure 7 It means that under the above parameter conditions, 10 users are randomly selected from ground users to analyze the task calculation delay of different algorithms. This method can meet the specified maximum tolerance delay of the task, while some tasks of DQN exceed the specified maximum tolerance delay of the task.

Claims

1. A method for optimizing the trajectory of unmanned aerial vehicles and offloading tasks for an air-ground integrated network, characterized in that: Establish network model, node communication model, task offloading energy consumption model and delay model; The optimization problem is to minimize the energy consumption and average delay of completing the task. The timing prediction algorithm and multi-agent deep reinforcement learning algorithm are used to solve the optimization problem and obtain the optimal trajectory optimization and task offloading strategy.

2. The method for unmanned aerial vehicle trajectory optimization and task offloading for an air-ground integrated network according to claim 1 is characterized in that: The network model is divided into three layers: ground layer, air layer and space layer; The ground layer includes ground users; The aerial layer is mainly composed of UAVs. UAVs establish communication links with ground users, receive tasks uploaded by ground users, and calculate the tasks themselves or pass them to upper-layer satellites for calculation based on decisions. At the same time, there are communication links between UAVs and satellites to achieve information transmission and interaction. The space layer includes multiple low-orbit satellites LEO in the sky; Deploy the iTransformer-based time series prediction model in it; The low-orbit satellite is regarded as a high-level intelligent agent, which makes high-level decisions at regular intervals based on the observed status information and the user distribution predicted by the iTransformer-based time series prediction model to guide the low-level strategy of the drone; The network model includes N ground users, M drones and L low-orbit satellites. The ground user set is represented as Its three-dimensional coordinates are expressed as l n (t) = (x n (t),y n (t),h n (t)), the coordinates of the ground users are randomly assigned at the beginning of each round, and then move randomly at the beginning of each time slot. The set of aerial drones is represented as Its three-dimensional coordinates are expressed as l m (t) = (x m (t),y m (t),h m (t)) is represented by moving at a speed V in each time slot and is responsible for collecting data from ground users. The set of low-orbit satellites above is represented by l∈L={1,2,...,L}, and its three-dimensional coordinates are represented by l l (t) = (x l (t),y l (t),h l (t)), LEO moves in a fixed orbit; ground users can establish communication with LEO satellites only when they enter the communication window.

3. The method for unmanned aerial vehicle trajectory optimization and task offloading for an air-ground integrated network according to claim 1 is characterized in that: In the node communication model, the communication link between the UAV and the ground user is a line-of-sight link LoS or a non-line-of-sight link NLoS; the path loss of Los is The path loss corresponding to NLoS is where β LoS and β NLoS are two different attenuation factors that distinguish LoS and NloS links, α mn (t) is the power gain, and the average path loss for the communication link is defined as: Among them, P LoS (t) is the probability of establishing a LoS channel between the UAV and the user in a specific network environment, and the probability of establishing a NLoS channel between the two is P NLoS (t) = 1-P LoS (t); Decouple the communication link between the UAV and the ground user, separate the uplink and downlink, so that each ground user can establish uplink and downlink communication with different UAVs, and define a binary variable To indicate whether UAVm is associated with user n, it is expressed as: When (·) = D, it indicates that the communication link is a downlink, and when (·) = U, it indicates that the link is an uplink. In each time slot, each user is only allowed to establish an uplink or downlink with one UAV. At the same time, due to the limitations of UAV hardware factors, the UAV can only establish connections with Q ground users at most in each time slot. Considering the channel interference during decoupling, the communication links between drones and between drones and satellites are established in line-of-sight transmission mode. Therefore, the information transmission rate between drones and between drones and satellites is defined as follows: Among them B back is the backhaul link bandwidth, σ 2 is the noise power, p o (t) is the transmission power; The information transmission rate between the UAV and the ground user is expressed as: is the link connection between drone m and user n at time t, B is the channel bandwidth, It is the signal to noise plus interference ratio. For uplink and downlink, the corresponding signal to noise ratio is: Among them, p n (t) represents the transmission power of user n, p m (t) represents the transmission power of UAV m, represents the interference caused by other users’ uplink transmission to UAV m, Indicates the interference caused by other drones’ downlink transmission to drone m, represents the self-interference of drone m, represents the interference caused by the downlink transmission of other drones to user n, represents the interference caused by other users' uplink transmission to user n, represents the self-interference of user n.

4. The method for unmanned aerial vehicle trajectory optimization and task offloading for an air-ground integrated network according to claim 1, characterized in that: When the task offloading energy consumption model is initialized, each ground user is randomly assigned a computing task with a certain amount of data. The ground user needs to upload the task to the UAV via the uplink, and then the UAV decides whether to upload the task to the LEO satellite for calculation or to calculate it itself. During the entire task calculation cycle, the energy consumed by the ground user and the UAV is mainly considered, which is specifically divided into four aspects: (1) Energy consumption of ground user data transmission: The user needs to transmit all his / her data to the UAV, and the data volume is I n , the power of user uploading data is defined as p n (t), so the transmission energy consumption is defined as: in Indicates the information transmission rate of the uplink; (2) Energy consumption of UAV data transmission: When the UAVs that establish uplink and downlink links with the same user are different, the UAV in the uplink transmits data to the corresponding UAV in the downlink, or forwards data to the LEO satellite, and its transmission power is defined as p m (t), the energy consumption is: The energy consumption of the UAV to perform calculations and return the results to the ground user when the task calculation is completed is: I m Indicates the amount of data that drone m needs to transmit, Indicates the downlink information transmission rate, Indicates the information transmission rate between drones or between drones and satellites; (3) Energy consumption of drone computing tasks: Assume that the amount of data required for the drone computing task is TA m , the calculation rate is Calculate the power as P c , the energy consumption of the computing task is defined as: (4) UAV flight energy consumption: The flight energy consumption of UAV is greater than that of hovering energy consumption, so the flight energy consumption of UAV is mainly considered. The flight power of UAV is defined as P f , the flight energy consumption is defined as:

5. The method for unmanned aerial vehicle trajectory optimization and task offloading for an air-ground integrated network according to claim 1, characterized in that: The delay model mainly includes two parts: task calculation delay and information transmission delay. The task calculation delay involves the UAV calculation delay and the satellite calculation delay, which are defined as: and The information transmission delay is divided into uplink delay, downlink delay, and transmission delay between drones and between drones and satellites, which are defined as: Among them, I n represents the size of the task uploaded by the ground user, I m Indicates the amount of data transmitted by the drone, I' m Indicates the amount of data transmitted between drones. Indicates the amount of data transmitted by the drone to the low-orbit satellite; Therefore, the cycle delay for each task to complete is defined as:

6. The method for unmanned aerial vehicle trajectory optimization and task offloading for an air-ground integrated network according to claim 1, characterized in that: The time series prediction algorithm divides the ground area into multiple small blocks, takes each small block as a unit, uses a real data set to simulate the number of ground users in the unit, and uses it as historical time series data to perform predictions through the iTransformer time series prediction model; The iTransformer-based time series prediction model is a Transformer-based time series prediction architecture. It does not change the Transformer network architecture, but transforms the roles of the attention mechanism and the feedforward network. It considers different variables separately, and each variable is encoded into an independent token. It uses the attention mechanism to model the correlation between different variables, and uses the feedforward network to model the time series correlation of variables to obtain sequence time series representation. It inputs historical time series data and outputs the number of users in each unit at future moments.

7. The method for unmanned aerial vehicle trajectory optimization and task offloading for an air-ground integrated network according to claim 1, characterized in that: The optimization problem is specifically: By optimizing the flight trajectory of the UAV and the offloading decision variables, the computational offloading constraint optimization problem is obtained with the goal of minimizing the total energy consumption of the system task and the weighted sum of the average delay, while ensuring the communication resource constraints, computing resource constraints, LEO satellite coverage constraints, and the maximum tolerable delay constraints of the task; the optimization problem is as follows: The constraints are: Among them, constraint C1 means that the power of UAV at any moment cannot exceed its maximum power; constraint C2 means the battery capacity limit of UAV, constraint C3 means the maximum coverage limit of UAV, constraint C4 means the tolerance delay of task calculation, constraint C5 means each user can only establish uplink or downlink connection with one UAV, constraint C6 means the flight speed limit of UAV, and constraint C7 means the instantaneous power consumption limit of user and UAV; the determined optimization problem is modeled as a partially observable Markov decision process POMDP, and the MDP is simplified and converted into a model-free process; the state space, action space and reward function are defined as follows; 1) State space: The state space includes all network factors that affect the agent's decision-making, expressed as Including the location coordinates of UAV, LEO, and ground users, the size of the mission, and energy consumption; 2) Action space: The action of UAV flying and the action of establishing connection with ground users are respectively represented as 3) Reward function: Aims to minimize the objective function, which is used as the reward function and is defined as:

8. The method for unmanned aerial vehicle trajectory optimization and task offloading for an air-ground integrated network according to claim 7, characterized in that: The multi-agent deep reinforcement learning algorithm is a two-level Qmix algorithm based on hierarchical learning; it includes two parts: a high-level decision network and a low-level decision network; Consider low-orbit satellites as high-level agents in a high-level decision-making network, and drones as low-level agents in a low-level decision-making network; The high-level decision network and the low-level decision network respectively include an agent's decision network RNN ​​and a mixing network for fitting multi-agent Q values; The RNN network loop inputs the current observation o and the action a of the agent at the previous moment t-1 , output Q value; the mixing network inputs the Q values ​​of multiple agents and the state information of the global environment, and outputs the global Q value Q tot ; An encoder based on a multi-head attention mechanism is added to the mixing network to reduce the dimension of the state information of the global environment, and the multi-head attention mechanism is used to extract effective spatiotemporal feature information. At the same time, the results predicted by the time series prediction algorithm are used as part of the reward function of the high-level decision network to guide the agent to identify areas with high returns in the early stage of strategy exploration and avoid the impact of sparse environmental rewards. The loss function of the mixing network is defined as: The mixing network adopts the network update method of DQN, θ represents the current network parameters, and b represents the number of samples; R is the reward function, θ - Represents the target network; the high-level decision network and the low-level decision network jointly learn the same task and share a target network to update network parameters; The high-level decision network is deployed on the low-orbit satellite, and the low-level decision network is deployed in the drone. The low-orbit satellite, as a high-level intelligent agent, makes a high-level decision every K time interval based on the observed state information and the user distribution predicted by the iTransformer-based time series prediction model to guide the drone's low-level strategy for the next k steps. The drone takes its own actions at each moment based on its own observation information and the reward function given by the high-level decision network in the low-orbit satellite. The reward value obtained by the high-level decision network is evenly distributed to the low-level decision network in the future k time slots. The reward function of the low-level decision network is expressed as: Where R H The reward function representing the high-level decision network is defined as: in, Indicates the number of users covered by the drone, It represents the difference between the predicted result of the time series prediction model based on iTransformer and the real data, and j and k represent their weights.

Images