Multi-unmanned aerial vehicle cooperative task unloading and trajectory optimization method

Abstract

The invention discloses a multi-unmanned aerial vehicle cooperative task unloading and trajectory optimization method, and relates to the field of vehicle networking and unmanned aerial vehicle cooperative computing. According to the invention, the space-time attention mechanism of Transform is combined with a multi-agent depth deterministic strategy algorithm framework, and intelligent collaboration and dynamic optimization of an unmanned aerial vehicle group are realized from three aspects of system architecture, feature modeling and decision strategy. The method is composed of four main stages: system architecture construction, feature modeling and state characterization, strategy generation and decision execution, and model training and parameter updating. A closed loop is formed among the four stages, and the whole-process collaboration from environment perception to intelligent decision-making to strategy optimization is realized step by step.

Description

Technical Field

[0001] This invention relates to the field of vehicle networking and drone collaborative computing, and in particular to a method for task offloading and trajectory optimization for multi-drone collaboration. Background Technology

[0002] With the rapid development of the Internet of Vehicles (IoV) and autonomous driving technologies, vehicles generate a large number of computationally intensive tasks during operation, such as environmental perception, path planning, video recognition, and safety control. These tasks have extremely high requirements for latency and reliability, but the computing power and energy of onboard terminals are limited, making it difficult for them to complete all tasks independently. To improve task processing efficiency, existing technologies typically employ task offloading, transferring some computational tasks to edge nodes or base stations with higher computing power for processing, thereby reducing the computational burden on the vehicle.

[0003] In traditional vehicle-to-everything (V2X) architectures, tasks are primarily offloaded and processed via ground base stations (BS) or roadside units (RSUs). Vehicles transmit task data wirelessly to the base station, which then performs calculations and returns the results. While this centralized approach can improve task processing speed to some extent, in environments with dense buildings or complex terrain, non-line-of-sight (NLoS) propagation easily occurs between vehicles and base stations, resulting in severe signal attenuation, degraded communication quality, and increased transmission latency. Furthermore, the fixed deployment locations of base stations limit their coverage area, making it difficult to dynamically adjust service range based on vehicle distribution, thus restricting overall system performance.

[0004] To overcome the above problems, existing solutions introduce unmanned aerial vehicles (UAVs) as mobile edge computing nodes. UAVs can establish line-of-sight (LoS) communication links in the air, and have advantages such as flexible deployment and wide coverage. UAVs can not only serve as aerial relay nodes to improve the quality of communication links, but also carry computing modules to directly undertake some computing tasks, realizing an air-ground collaborative task processing mode. At present, research on UAV-assisted vehicle networking mainly focuses on two directions: (1) UAV trajectory optimization: by planning the flight path of UAVs, they can maintain a high-quality connection with vehicles, improve communication speed and reduce flight energy consumption; (2) Task offloading and resource allocation optimization: rationally allocate communication and computing resources among vehicles, UAVs and base stations to reduce the overall system latency and energy consumption.

[0005] Existing typical solutions often employ the Multi-Agent Reinforcement Learning (MARL) framework. These methods typically adopt a "Centralized Training and Decentralized Execution" (CTDE) structure, treating each UAV as an independent agent and learning optimal unloading and flight strategies through deep reinforcement learning algorithms. Among these, the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm is a relatively mature approach. Some improved solutions introduce dual-criteria networks (such as MATD3) into the MADDPG framework to mitigate Q-value overestimation, or combine imitation learning with pre-training mechanisms to improve convergence speed. These methods can achieve collaborative learning and joint optimization among UAV swarms in simulation environments.

[0006] Overall, this type of method achieves joint optimization of task unloading and trajectory through collaborative learning among intelligent agents, and demonstrates good adaptability and optimization performance in complex dynamic environments.

[0007] Based on the above description, although existing multi-agent deep reinforcement learning methods (such as MADDPG and MATD3) have achieved certain results in UAV cooperative task offloading and trajectory optimization, they still have the following drawbacks:

[0008] 1) Lack of spatiotemporal feature modeling for task generation: Task requests in the connected vehicle environment have significant spatiotemporal dynamic characteristics, such as the tendency to form task "hotspots" during peak hours or in specific areas. Existing methods typically rely on fully connected neural networks to model the state, which makes it difficult to capture the correlation between vehicle tasks in spatial distribution and temporal evolution, resulting in insufficient prediction and response of UAVs to task load.

[0009] 2) Limited global collaboration efficiency among drones: In multi-drone scenarios, there are problems such as resource competition and overlapping coverage areas among individuals, and optimization requires global coordination capabilities. However, traditional methods such as MADDPG lack an effective global feature extraction mechanism during policy update. Each drone agent mainly relies on local observations for decision-making, which often leads to getting trapped in local optima and makes it difficult to achieve global optimization at the group level.

[0010] 3) Insufficient convergence speed and environmental adaptability: The connected vehicle scenario is dynamic and complex, with rapid changes in vehicle movement and task distribution. Existing methods typically require a large number of iterations during training to reach a stable policy, and are prone to performance degradation or even training instability when the task load fluctuates drastically, which limits their application in real-world scenarios.

[0011] To address the aforementioned problems, the present invention aims to provide a method for task offloading and trajectory optimization in multi-UAV collaboration. By combining the spatiotemporal attention mechanism of Transformer with the multi-agent deep deterministic policy algorithm framework, intelligent collaboration and dynamic optimization of UAV swarms are achieved from three levels: system architecture, feature modeling, and decision-making strategy. Summary of the Invention

[0012] This invention proposes a method for task unloading and trajectory optimization in multi-UAV collaborative operation. Figure 1 The flowchart of the method of this invention is shown below. The method consists of four main stages: system architecture construction, feature modeling and state representation, policy generation and decision execution, and model training and parameter updating. These four stages form a closed loop, gradually realizing the coordinated operation of the entire process from environmental perception to intelligent decision-making and then to policy optimization. The operation flow of this invention is as follows:

[0013] S1: System Architecture Construction: In this stage, the invention first establishes a heterogeneous edge collaborative architecture integrating vehicle, air, and ground. The system mainly consists of vehicle terminals, UAV nodes, and ground base stations, which collaborate to complete task perception, offloading, and computation. In this stage, the system uniformly models the computing power, communication bandwidth, and energy consumption of each node, and establishes a two-layer communication link structure between the vehicle, UAV, and base station. Simultaneously, to support subsequent algorithm operation, the invention includes an architecture modeling module. This module is responsible for defining task flow paths, node topology, and resource allocation relationships, providing fundamental support for feature modeling and policy optimization. Through the above modeling process, the system achieves a unified description of computing resources and communication networks, forming a scalable collaborative optimization architecture, laying the foundation for subsequent spatiotemporal feature extraction and policy training.

[0014] S2: Feature Modeling and State Representation: The system extracts task features and generates a global state based on the collected real-time data (feature modeling module). This step is mainly deployed on the UAV side and maintains data interaction with the vehicle terminal. The module uses the Transformer coding structure to model the spatiotemporal and data features of the vehicle task, integrating spatial location coding, temporal dynamic coding, and data feature coding. It utilizes a multi-head attention mechanism to extract the global correlation and dynamic change patterns of the vehicle task distribution. This generates a global state representation reflecting the migration of task hotspots and the vehicle's movement trend, providing a global perspective input for subsequent strategy generation.

[0015] S3: Policy Generation and Decision Execution: This step is primarily deployed in the intelligent decision-making unit of the UAV. During the training phase, the system employs a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) framework for joint optimization using centralized information. The module concatenates the UAV's local observations and global state representations and inputs them into the Actor network, outputting continuous actions such as task unloading ratio and flight trajectory adjustment. During the centralized training phase, the Critic network takes the global state and the joint actions of all UAVs as input, evaluates the action value function, and guides Actor policy updates. In the actual operation phase, each UAV independently executes task unloading and trajectory adjustment operations based on the output of the policy generation module, while simultaneously transmitting system state, execution decisions, system rewards, and next state data back to the cloud training platform for experience playback and parameter updates, achieving a combination of distributed execution and centralized optimization.

[0016] S4: Model Training and Parameter Update: This stage is primarily completed on the cloud server side, used for centralized training and parameter optimization of all agent policy networks, enabling continuous model iteration and performance improvement. During training, the system utilizes an experience replay mechanism to store samples generated by the interaction between the drone and the environment (including states, actions, rewards, and next states), and uses mini-batch sampling for batch updates. Through the centralized training mechanism, the cloud can uniformly evaluate the joint actions of each drone, thereby learning a stable and efficient global policy. After training is complete, the optimized model parameters are distributed to edge nodes, which are then invoked by the drones in the distributed execution phase, achieving continuous online optimization under the centralized training and distributed execution (CTDE) mechanism.

[0017] Through the organic collaboration of the above steps and modules, this invention can realize intelligent collaboration of drone swarms in dynamic vehicle networking environments, significantly reducing system latency and energy consumption, and improving overall service quality and network reliability.

[0018] 1) This invention designs a heterogeneous edge collaborative architecture for UAV-assisted vehicle-to-everything (V2X) optimization. This method models both the UAV and the base station as edge computing nodes. The UAV not only acts as a relay node to establish line-of-sight communication links in the air to improve data transmission rate and reliability, but also serves as a mobile edge computing unit to handle some vehicle task computations. Through this architecture, vehicles can flexibly choose unloading paths based on network conditions and task requirements, thereby achieving collaborative division of labor between ground base stations and aerial UAVs to improve computing power and communication efficiency in complex V2X environments.

[0019] 2) This invention designs a spatiotemporal feature modeling method combining Transformer. By introducing spatial location encoding, temporal dynamic encoding, and data feature encoding during the state modeling process, the UAV can capture the distribution patterns of vehicle tasks in different regions and at different times. Combined with a multi-head self-attention mechanism, a global spatiotemporal representation is generated. This method not only reflects the spatial concentration and temporal evolution trends of task load but also provides the UAV with global task hotspot migration information, thereby improving the UAV's perception and prediction capabilities in dynamic vehicle-to-everything (V2X) environments.

[0020] 3) This invention designs a reinforcement learning framework that integrates Transformer and Multi-Agent Deep Deterministic Policy Gradient (MADDPG). In this framework, the Actor network input consists of a concatenation of local observations and global spatiotemporal representations, used to output the task offloading ratio and flight trajectory adjustment actions of the UAV; the Critic network, during the centralized training phase, inputs the global state and the joint actions of all UAVs to evaluate the action value function. This design maintains the flexibility of distributed execution of UAVs while achieving efficient collaboration among multiple UAVs through global information modeling, thereby improving the overall performance of joint task offloading and trajectory optimization.

[0021] 4) This invention proposes a reward function for joint optimization of task unloading and trajectory. The reward function takes the weighted sum of the overall system latency and energy consumption as the optimization objective, and introduces safety buffer constraints between UAVs and buildings and minimum spacing constraints between UAVs. This enables UAVs to improve the overall efficiency of task completion while avoiding collisions and excessive energy consumption, thereby ensuring that the vehicle network system can achieve low latency, high energy efficiency and safe and reliable operation in dynamic environments. Attached Figure Description

[0022] Figure 1 Flowchart of multi-UAV collaborative task offloading and trajectory optimization method

[0023] Figure 2 A hierarchical architecture diagram of a multi-UAV collaborative task offloading and trajectory optimization method.

[0024] Figure 3 Flowchart of Spatiotemporal Feature Encoding in Transformer

[0025] Figure 4 Schematic diagram of a multi-agent reinforcement learning training framework

[0026] Figure 5 Training flowchart of the MADDPG algorithm Detailed Implementation

[0027] 1. Layered Framework Description

[0028] 1.1 Terminal Device Layer: The terminal device layer mainly consists of vehicle terminals and is the source of task generation. During the operation of the vehicle-to-everything (V2X) network, vehicles continuously generate computationally intensive tasks, including environmental perception, path planning, and video analysis. These tasks are first generated within the onboard terminal and are initially marked and reported according to task size and computational requirements. The terminal device layer is also responsible for uploading real-time vehicle motion status information, such as location, speed, and surrounding environment perception data, providing input support for subsequent task offloading and resource scheduling. Vehicle terminals can directly execute some lightweight tasks when necessary, while tasks with high computational requirements or strict latency requirements need to be offloaded to the upper layer for processing via the communication link.

[0029] 1.2 Edge Server Layer: The edge server layer, mainly composed of UAV nodes and ground base stations, is the core computing and collaboration layer of this invention. In this layer, UAVs establish airborne communication links through flight trajectory adjustments to alleviate non-line-of-sight propagation problems between vehicles and base stations. Simultaneously, UAVs carry computing resources to act as mobile edge computing nodes, providing task processing capabilities for vehicles. The edge server layer deploys two key functional modules proposed in this invention: an architecture modeling module and a feature modeling module. The former is responsible for constructing a vehicle-air-ground integrated heterogeneous edge architecture, clarifying the topological relationships between computing and communication resources; the latter models the spatiotemporal distribution of vehicle tasks using a Transformer model, generating a global state representation to provide input for UAV swarm decision-making. Finally, this layer calls the MADDPG reinforcement learning framework through the policy generation module to jointly output the task offloading ratio and UAV trajectory adjustment scheme, thereby achieving intelligent collaboration among multiple UAVs.

[0030] 1.3 Cloud Server Layer: The cloud server layer, serving as the core of global optimization and parameter management, primarily consists of a central controller and a training platform. It receives state parameters, action decisions, and reward feedback from the edge server layer and centrally trains strategies to update the parameters of the Transformer-MADDPG model. Simultaneously, the cloud server layer distributes the latest trained model parameters to the edge server layer to guide the drone swarm in achieving efficient decision-making during the distributed execution phase. Through this centralized training and distributed execution mechanism, the cloud server layer achieves global parameter synchronization and policy consistency for the algorithm, enhancing the adaptability of the drone swarm in dynamic vehicle-to-everything (V2X) environments.

[0031] In summary, this invention achieves a complete closed loop through a layered framework, from task generation and reporting at the vehicle terminal, to collaborative optimization processing at the edge server layer, and finally to centralized training and parameter distribution at the cloud server layer. This layered structure combines the resource characteristics of vehicles, drones, and base stations to improve task processing efficiency and system reliability in a vehicle-to-everything (V2X) environment. The following section will introduce the key functional modules.

[0032] 2. Introduction to Key Module Functions

[0033] After completing the overall design of the layered framework, this invention further proposes three key functional modules to achieve efficient collaboration and joint optimization among multiple UAVs in a vehicle-to-everything (V2X) environment. These three modules are the architecture modeling module, the feature modeling module, and the policy generation module. They work together to form a complete technology chain from system resource abstraction and task feature modeling to intelligent decision generation. The specific functions are described below:

[0034] 2.1 Architecture Modeling Module

[0035] (1) System architecture generation:

[0036] The architecture modeling module first generates the overall architecture of the vehicle-to-everything (V2X) network. In this stage, the invention integrates vehicles, drones, and ground base stations into a heterogeneous edge collaborative architecture, clearly defining the roles of various nodes in computing and communication. Vehicles primarily act as task initiators, generating task requests of varying scales and computational complexity; drones in the air can act as relay nodes to establish line-of-sight communication links and also as mobile edge nodes with computing capabilities; base stations serve as core ground computing units, possessing stronger computing power and stable communication capabilities. Through this architecture generation process, a system operating framework for multi-drone collaborative optimization is formed, enabling task processing and resource scheduling to proceed in an orderly manner within the hierarchical architecture.

[0037] (2) System node modeling:

[0038] Based on the generated overall architecture, the architecture modeling module further models different types of nodes in the system. The task of vehicle node n is determined by the task size L. n (t) and the number of cycles required for calculation C n (t) is quantized. The vehicle's position and motion state are quantized using coordinates q. n (t)=[x n (t),y n (t)] and velocity direction tuple [V n (t),θ n [(t)] represents the drone node, which is determined by its flight position q. u (t)=[x u (t),y u (t),Z u [t], Task queue length K u (t), computing resources f u Spectrum Resources B u Isoparametric characterization; base station nodes, on the other hand, focus on their fixed computing resources f. u and spectrum resources B bModeling. By establishing these node models, the distribution of computing and communication resources in the system can be accurately described, providing constraints and optimization objectives for subsequent task offloading and trajectory planning. System node modeling provides a realistic and comprehensive environmental foundation for the reinforcement learning process, thereby improving the feasibility and applicability of the generation strategy.

[0039] (3) Communication link construction:

[0040] After completing the system architecture and node modeling, the architecture modeling module is also responsible for constructing the integrated vehicle-air-ground communication link. The communication link in this invention is divided into two categories: vehicle-UAV and UAV-base station. The communication link between the vehicle and the UAV mainly relies on line-of-sight propagation. This invention fixes the UAV's flight altitude; therefore, the three-dimensional distance between the vehicle and the UAV can be expressed as:

[0041]

[0042] Its horizontal projection distance is:

[0043]

[0044] Considering that there is always line-of-sight propagation between the drone and the vehicle, the communication adopts a free-space path loss model, therefore h is used. nu (t) represents the channel gain between the vehicle and the drone:

[0045]

[0046] h0 is the channel gain reference value, corresponding to a distance of 1m and a transmit power of 1W / hour. The transmit power of vehicle n is P. n According to Shannon's formula: at time t, the uplink transmission rate R between vehicle n and drone u is... nu (t) is:

[0047]

[0048] Where B is the channel bandwidth between the vehicle and the drone, expressed as:

[0049] B = B u / N u (t)#(5)

[0050] B u It is the bandwidth of the drone, N u (t) represents the total number of tasks unloaded to the UAV from vehicles within its range at time t. This is the noise power spectral density. For ease of calculation, we assume that the bandwidth of the UAV is uniformly allocated to each task.

[0051] The backhaul link between the drone and the base station relies on stable air-to-ground transmission conditions to ensure that the unloading task can be efficiently transmitted to the base station for processing. The coordinates of the base station are defined as: q b (t), denoted as [x b (t),y b (t),Z b [t], the three-dimensional distance d between the drone and the base station ub (t) is:

[0052]

[0053] Since the communication between the drone and the base station is also LOS (Local Optical System), the channel gain h between the drone and the base station is... ub (t) is defined as:

[0054]

[0055] The drone's transmission power is P u At time t, the number of tasks that UAV u unloads from base station b is N. b (t), It is the noise power spectral density, and the channel bandwidth between the UAV and the base station is B. b Therefore, the downlink transmission power R ub (t) is:

[0056]

[0057] This communication link model is used to characterize the transmission characteristics and latency parameters under different offloading paths, providing input data for task scheduling and strategy optimization.

[0058] 2.2 Feature Modeling Module

[0059] (1) Real-time environmental status monitoring: The feature modeling module first performs real-time status monitoring of the vehicle-to-everything (V2X) environment. The UAV obtains the vehicle's operating status information through continuous communication with the vehicle terminal and base station, thereby obtaining local observations from the UAV. u (t) value. Where:

[0060]

[0061] Where N is the sum of all vehicle nodes. This real-time information not only describes the generation of vehicle tasks but also reflects the dynamic changes in the environment. Through continuous acquisition and updating of environmental conditions, the UAV can form a perception basis for task distribution and network load, thus providing raw data support for subsequent task feature modeling. This function is used to maintain the real-time performance and accuracy of system input to support the UAV's response to task changes and communication conditions in the vehicle-to-everything (V2X) environment, providing real-time data for subsequent Transformer coding.

[0062] (2) Spatiotemporal Feature Encoding: After completing the real-time acquisition of vehicle task and environmental status, this invention performs deep modeling of multi-source information through a joint encoding mechanism of spatiotemporal and task data features. The core idea of ​​this module is to leverage the advantages of the Transformer architecture in capturing global dependencies, while integrating the spatial distribution features, temporal dynamic evolution features, and attribute features of the vehicle task itself, thereby generating a high-dimensional and more expressive feature vector as input for subsequent strategy generation.

[0063] In the spatial dimension, a location encoding mechanism is employed to model the geometric relationship between the vehicle and the drone. By converting the vehicle's position, the drone's current position, and relative distance into vectorized information, the system uses location encoding to identify the spatial distribution of tasks in different areas, thereby determining the clustering and migration of task hotspots. This spatial location encoding ensures that the model can distinguish task requests from different geographical locations, thus providing support for drone trajectory planning and coverage optimization.

[0064] In the time dimension, a dynamic temporal coding method is introduced to characterize the temporal patterns of task generation. Vehicle tasks typically exhibit both suddenness and periodicity; for example, task arrival rates increase significantly during peak traffic hours. By performing time series modeling of task arrival rates, queue evolution trends, and their coupling relationship with vehicle speed, the system can predict future trends in task demand, enhancing the UAV's forward-looking decision-making capabilities.

[0065] At the task feature level, the data size and required CPU cycles for each task are further encoded into vectors. Task feature encoding can clearly define the computational complexity and service requirements of different tasks, enabling the subsequent policy generation process to better balance latency and energy consumption when allocating tasks and scheduling resources.

[0066] Through a multi-head self-attention mechanism, spatiotemporal encoding and task feature encoding are jointly processed. The model can simultaneously focus on spatial proximity, temporal continuity, and task complexity globally, extracting implicit correlations between different dimensions. Finally, the system outputs a fused high-dimensional global feature representation to enhance the environmental perception and task prediction capabilities of UAVs in complex vehicle-to-everything (V2X) environments. The encoding process is as follows: Figure 3 : Where PE, TE, and AE represent location encoding, time encoding, and task data feature encoding, respectively.

[0067] (3) Global State Representation: After spatiotemporal encoding, the feature modeling module fuses multi-source information to generate a global state representation. This representation includes not only the spatiotemporal distribution characteristics of vehicle tasks but also the relative positional relationships between UAVs, thus forming a complete system-level feature vector. The global state representation provides input for the subsequent policy generation module, enabling each UAV to make more reasonable task unloading and trajectory adjustment decisions by leveraging the state representation containing global information, rather than relying solely on local observations. This function significantly enhances the environmental perception and prediction capabilities of UAVs, enabling group collaboration to better cope with complex scenarios such as task hotspot migration and rapid changes in vehicle distribution, thereby achieving global performance optimization.

[0068] 2.3 Strategy Generation Module

[0069] (1) State Vector Fusion: The policy generation module first completes the fusion of state vectors. This process integrates the local observation information of the UAV. u The vector o' is concatenated with the global state representation s'(t) generated by the feature modeling module to form the complete input state vector of the UAV u. u (t) is represented as:

[0070] o' u (t)=Concat(o u (t),s'(t))#(10)

[0071] Through a feature fusion mechanism, this invention achieves complementarity between global and local information at the input level, enabling UAVs to consider both local execution constraints and the overall global optimization objective when performing task unloading and trajectory planning. The state vector fusion result provides a high-dimensional, information-complete input for the subsequent decision network, effectively enhancing the collaborative capabilities and environmental adaptability among multiple UAVs.

[0072] (2) MADDPG agent decision-making: At time t, each UAV acts as an independent agent, utilizing the input vector o' of the Actor network. u (t) Generates continuous action a u (t), which includes the task offloading ratio (local execution, offloading to a base station or other drones) and flight trajectory adjustment variables (speed and direction), can be specifically expressed as:

[0073]

[0074] in, This is the decision result of the MATD3 agent. N represents the Actor network parameters and its weights. tDuring the training phase, exploratory noise is injected into the drone's decision-making actions to promote policy exploration and prevent the model from getting trapped in local optima too early.

[0075] During the decision-making phase, each UAV independently completes its action output based on the obtained optimal strategy parameters and executes them in a distributed manner in the real environment, thereby achieving collaborative operation and task distribution among UAVs. This mechanism unifies and optimizes the strategy model, achieving overall coordination during distributed execution to reduce resource conflicts and trajectory interference. Ultimately, the optimized actions output by the strategy generation module can effectively reduce task processing latency and energy consumption in a dynamic vehicle-to-everything (V2X) environment, improving the overall system performance.

[0076] 2.4 Algorithm Training Module

[0077] Algorithm training is completed on a cloud server, centrally training the Actor-Critic network for all agents. During the training phase, an experience replay pool stores state-action-reward samples from heterogeneous environments, and policy parameters are soft-updated through the target network to improve stability. This module performs centralized training and parameter updates on the Transformer-MADDPG model based on data uploaded from the edge server layer and reward feedback, achieving policy consistency and global performance optimization. In this invention, the network weight parameters involved in the Transformer network, the Actor network, and the Critic network in multi-agent reinforcement learning are automatically learned through the centralized training process, rather than being fixed values ​​preset by humans.

[0078] Specifically, during the training phase, the cloud server continuously interacts with the vehicle-to-everything (V2X) simulation environment, iterating and updating the model multiple times based on an experience playback mechanism. In each iteration, the system updates the policy network parameters using gradients based on the reward function, gradually stabilizing the UAV mission unloading decision and trajectory adjustment strategies. The model is considered to have reached convergence when the number of training iterations reaches a preset threshold, or when the change in the system's objective function is less than a set threshold.

[0079] After model convergence, the Transformer network parameters and the policy parameters of the Actor and Critic networks are fixed and distributed to the UAV side. During actual operation, they are only used for forward inference calculations and no longer participate in the parameter update process. Furthermore, training control parameters such as the learning rate, soft update coefficient, and exploration noise intensity involved in the training process can be set offline according to different vehicle-to-everything (V2X) application scenarios. Their specific values ​​do not constitute a limitation on the technical solution of this invention and are conventional technical means that can be flexibly adjusted by those skilled in the art according to actual needs.

[0080] Multi-agent reinforcement learning training framework, such as Figure 4As shown:

[0081] The objective of this invention is to minimize the sum of the weighted average latency and energy consumption of all UAVs, thereby achieving a balance between real-time performance and energy efficiency in complex and dynamic vehicle-to-everything (V2X) environments. This objective is defined as an optimization problem:

[0082]

[0083] Where q u (t) is the weighted average of the delay and energy consumption of the UAV u at time t, expressed as:

[0084]

[0085] Where ω1 and ω2 are weighting coefficients for latency and energy consumption, ω1 + ω2 = 1. Their specific values ​​depend on the specific requirements of the vehicle-to-everything (V2X) scenario (i.e., the emphasis on latency and energy consumption), rather than a fixed solution.

[0086] The training process of the MADDPG algorithm is as follows: Figure 5 At time t With Gaussian noise N t Combined, we get a u (t) is applied to the current vehicle-to-everything (V2X) environment to obtain the reward function r of the drone u. u (t), In order to maximize the reward function and be relevant to the purpose of this invention, the reward function r of the drone u is set. u (t) is:

[0087] r u (t)=-lnq u (t)#(14)

[0088]

[0089] Global information includes state space and action space: state space s i (t) represents all intelligent agents o' u The set of action values ​​(t), where action space A(t) is the action value a of the agent. u The set of (t) is represented as follows:

[0090] s i (t)={o'1(t),……o' U (t)}#(16)

[0091] A(t)={a1(t)…a U (t)}#(17)

[0092] a u(t) Input into the Critic evaluation network, the Critic network calculates the Q value by combining global information (through state sharing among UAVs), where the action value function Q output by UAV u is... u (t) is represented as:

[0093]

[0094] in These are parameters of the Critic network, representing its weights, and do not have fixed, specific values. They are internal weight parameters obtained through continuous learning and optimization during training, and their final values ​​depend on the training data, environmental interactions, and the model's convergence state.

[0095] The drone will a u (t) is applied to the current vehicle-to-everything (V2X) environment to obtain the local observation stitched value o' for the next state. u (t+1) and input it into the Actor's target network to obtain the action value a for the next state. u (t+1), and obtain the Q value of the next state through the Critic target network. The calculation formula is as follows:

[0096]

[0097] in These are the target Critic network parameters and weights. The Critic network updates its evaluation network parameters by minimizing a loss function. As shown in formula (6):

[0098]

[0099] Subsequently, the Actor evaluation network parameters are updated by maximizing the policy objective function. The formula for updating the Actor network policy gradient is:

[0100]

[0101] in, Let be the policy objective function, which represents the objective function when the parameter is... The expected long-term cumulative reward under the strategy.

[0102] The formula for calculating the policy gradient is:

[0103]

[0104] in, This represents the gradient of the Actor network with respect to its own parameters. Let be the gradient of the Q-value function of the Critic network with respect to the action of agent u.

[0105] When evaluating parameters in a network and Target network parameters during continuous updates and Update using a soft update method:

[0106]

[0107] Where τ is the soft update coefficient, which is a constant between 0 and 1, and is usually determined by offline hyperparameter tuning experiments to a small fixed value (such as 0.01 or 0.05).

Claims

1. A method for task unloading and trajectory optimization through multi-UAV collaboration, characterized in that: S1: System Architecture Construction: First, a heterogeneous edge collaborative system integrating vehicle, air, and ground is established. The system mainly consists of vehicle terminals, UAV nodes, and ground base stations, which work together to complete task perception, offloading, and computation. In this stage, the system uniformly models the computing power, communication bandwidth, and energy consumption of each node, and establishes a two-layer communication link structure between the vehicle, UAV, and base station. At the same time, to support the subsequent algorithm operation, an architecture modeling module is set up in the system. This module is responsible for defining the task flow path, node topology, and resource allocation relationship, providing basic support for feature modeling and strategy optimization. S2: Feature Modeling and State Representation: The system's feature modeling module extracts task features and generates a global state based on the collected real-time data. Deployed on the UAV side, it maintains data interaction with the vehicle terminal. The module uses a Transformer coding structure to model the spatiotemporal features of the vehicle task, integrating spatial location coding and temporal dynamic coding. It utilizes a multi-head attention mechanism to extract the global correlation and dynamic change patterns of the vehicle task distribution. This generates a global state representation that reflects the migration of task hotspots and the vehicle's movement trend, providing a global perspective input for subsequent strategy generation. S3: Strategy Generation and Decision Execution: Deployed in the intelligent decision-making unit of the UAV; during the training phase, the system adopts a multi-agent deep deterministic policy gradient framework to perform joint optimization with centralized information; the module inputs the UAV's local observations and global state representations into the Actor network, outputting continuous actions such as task unloading ratio and flight trajectory adjustment; during the centralized training phase, the Critic network takes the global state and the joint actions of all UAVs as input, evaluates the action value function, and guides the Actor policy update; during the actual operation phase, each UAV independently executes task unloading and trajectory adjustment operations according to the output of the strategy generation module, while transmitting the task completion rate, energy consumption, and location information back to the cloud training platform for experience playback and parameter updates, realizing the combination of distributed execution and centralized optimization; S4: Model training and parameter update: Completed on the cloud server. During the training process, the system uses an experience replay mechanism to store samples generated by the interaction between the drone and the environment, including state, action, reward and next state, and uses batch sampling to update in batches. After training is complete, the optimized model parameters will be distributed to edge nodes and invoked by drones during the distributed execution phase, achieving continuous online optimization under the centralized training and distributed execution mechanism.

2. The method according to claim 1, characterized in that: 2.1 Architecture Modeling Module (1) System architecture generation: The architecture modeling module first generates the overall architecture of the vehicle-to-everything (V2X) network. In this stage, vehicles, drones, and ground base stations are integrated into a heterogeneous edge collaborative architecture, clarifying the roles of various nodes in computing and communication. Vehicles mainly serve as the initiator of tasks, generating task requests of different scales and computational complexities. Drones can act as relay nodes to establish line-of-sight communication links and as mobile edge nodes with computing capabilities. Base stations serve as the core ground computing units. Through this architecture generation process, a system operation framework for multi-drone collaborative optimization is formed, enabling task processing and resource scheduling to proceed in an orderly manner under the hierarchical architecture. (2) System node modeling: Based on the generated overall architecture, the architecture modeling module further models different types of nodes in the system; the task of vehicle node n is determined by the task size L. n (t) and the number of cycles required for calculation C n (t) is quantized; the vehicle's position and motion state are respectively represented by coordinates q. n (t)=[x n (t),y n (t)] and velocity direction tuple [V n (t),θ n [t] represents the drone node, which is determined by its flight position q. u (t)=[x u (t),y u (t),Z u [t], Task queue length K u (t), computing resources f u Spectrum Resources B u Isoparametric characterization; Base station nodes, on the other hand, focus on their fixed computing resources f u and spectrum resources B b Modeling; (3) Communication link construction: After completing the system architecture and node modeling, the architecture modeling module is also responsible for constructing the integrated vehicle-air-ground communication link. This communication link is divided into two categories: vehicle-to-drone and drone-to-base station. The communication link between the vehicle and the drone mainly relies on line-of-sight propagation, with a fixed drone flight altitude. Therefore, the three-dimensional distance between the vehicle and the drone can be represented as: Its horizontal projection distance is: Considering that there is always line-of-sight propagation between the drone and the vehicle, the communication adopts a free-space path loss model, therefore h is used. nu (t) represents the channel gain between the vehicle and the drone: h0 is the channel gain reference value, corresponding to a distance of 1m and a transmit power of 1W / hour; the transmit power of vehicle n is P. n According to Shannon's formula: at time t, the uplink transmission rate R between vehicle n and drone u is... nu (t) is: Where B is the channel bandwidth between the vehicle and the drone, expressed as: B=B u / N u (t)#(5) B u It is the bandwidth of the drone, N u (t) represents the total number of tasks unloaded to the UAV from vehicles within its range at time t; This is the noise power spectral density; for ease of calculation, it is assumed that the bandwidth of the UAV is uniformly allocated to each task. The backhaul link between the drone and the base station relies on stable air-to-ground transmission conditions to ensure that the unloading task can be efficiently transmitted to the base station for processing; the coordinates of the base station are defined as: q b (t), denoted as [x b (t),y b (t),Z b [t], the three-dimensional distance d between the drone and the base station ub (t) is: Since the communication between the drone and the base station is also LOS (Local Optical System), the channel gain h between the drone and the base station is... ub (t) is defined as: The drone's transmission power is P u At time t, the number of tasks that UAV u unloads from base station b is N. b (t), It is the noise power spectral density, and the channel bandwidth between the UAV and the base station is B. b Therefore, the downlink transmission power R ub (t) is: This communication link model is used to characterize the transmission characteristics and latency parameters under different offloading paths, providing input data for task scheduling and strategy optimization.

3. The method according to claim 1, characterized in that: 2.2 Feature Modeling Module (1) Real-time environmental status monitoring: The feature modeling module first performs real-time status monitoring of the vehicle network environment; the UAV obtains the vehicle's operating status information through continuous communication with the vehicle terminal and base station, thereby obtaining the UAV's local observations. u (t) value; where: Where N is the sum of all vehicle nodes; this real-time information is not only used to describe the generation of vehicle tasks, but also reflects the dynamic changes in the environment; through continuous collection and updating of environmental status, the UAV can form a perception basis for task distribution and network load, thereby providing raw data support for subsequent task feature modeling; this function is used to maintain the real-time and accuracy of system input to support the UAV's response to task changes and communication conditions in the vehicle network environment, and provides real-time data for subsequent Transformer coding; (2) Spatiotemporal feature encoding: After real-time acquisition of vehicle task and environmental status, multi-source information is deeply modeled through spatiotemporal and task feature joint encoding mechanism; this module utilizes the advantages of Transformer architecture in capturing global dependencies, and integrates the spatial distribution features, temporal dynamic evolution features and attribute features of vehicle task itself. In the spatial dimension, a position coding mechanism is used to model the geometric relationship between the vehicle and the drone. By converting the vehicle position, the current position of the drone and the relative distance into vectorized information, the system identifies the spatial distribution of tasks in different areas through position coding, which is used to determine the clustering and migration of task hotspots. In the time dimension, a dynamic time coding method is introduced to characterize the temporal pattern of task generation. Vehicle tasks are usually sudden and periodic. By performing time series modeling on task arrival rate, queue evolution trend and its coupling relationship with vehicle speed, the system predicts the changing trend of future task demand. In terms of task characteristics, the data size and required CPU cycles for each task are further encoded into vectors. Through a multi-head self-attention mechanism, spatiotemporal encoding and task feature encoding are jointly processed. The model can simultaneously focus on spatial proximity, temporal continuity and task complexity in a global scope, and extract the implicit correlations between different dimensions. Finally, the system outputs a fused high-dimensional global feature representation to enhance the environmental perception and task prediction capabilities of UAVs in complex vehicle-to-everything (V2X) environments. (3) Global state representation: After the spatiotemporal encoding is completed, the feature modeling module will fuse multi-source information to generate a global state representation. This representation includes not only the spatiotemporal distribution characteristics of vehicle tasks, but also the relative positional relationships between UAVs, thus forming a complete system-level feature vector. The global state representation provides input for the subsequent policy generation module.

4. The method according to claim 1, characterized in that: 2.3 Strategy Generation Module (1) State Vector Fusion: The policy generation module first completes the fusion of state vectors; this process integrates the local observation information of the UAV. u The vector o'(t) is concatenated with the global state representation s'(t) generated by the feature modeling module to form the complete input state vector of the UAV u; the concatenated vector o' u (t) is represented as: o' u (t)=Concat(o u (t),s'(t))#(10) Through feature fusion mechanism, the complementarity of global and local information is achieved at the input level, enabling UAVs to consider both local execution constraints and the overall global optimization goal when performing task unloading and trajectory planning. The result of state vector fusion provides high-dimensional and information-complete input for the subsequent decision network, effectively enhancing the collaboration capability and environmental adaptability among multiple UAVs. (2) MADDPG agent decision-making: At time t, each UAV acts as an independent agent, utilizing the input vector o' of the Actor network. u (t) Generates continuous action a u (t), which includes the task offloading ratio (local execution, offloading to a base station or other drones) and flight trajectory adjustment variables (speed and direction), can be specifically expressed as: in, This is the decision result of the MATD3 agent; N represents the Actor network parameters, indicating the Actor network weights. t During the training phase, exploratory noise is injected into the drone's decision-making actions to promote policy exploration and prevent the model from getting trapped in local optima too early.

5. The method according to claim 1, characterized in that: 2.4 Algorithm Training Module Algorithm training is completed on a cloud server, where the Actor-Critic network of all agents is trained centrally. During the training phase, an experience replay pool is used to store state-action-reward samples in heterogeneous environments, and the policy parameters are updated softly through the target network to improve stability. This module performs centralized training and parameter updates on the Transformer-MADDPG model based on the data uploaded from the edge server layer and the reward feedback to achieve policy consistency and global performance optimization. During implementation, the network weight parameters involved in the Transformer network, the Actor network and the Critic network in multi-agent reinforcement learning are all automatically learned through a centralized training process. During the training phase, the cloud server continuously interacts with the vehicle networking simulation environment and iterates and updates the model multiple times based on the experience playback mechanism. In each iteration, the system updates the policy network parameters according to the reward function, so that the UAV mission unloading decision and trajectory adjustment strategy gradually become stable. After the model converges, the Transformer network parameters, as well as the policy parameters of the Actor and Critic networks, are fixed and distributed to the UAV side.

6. The method according to claim 1, characterized in that: Define the objective as an optimization problem: Where q u (t) is the weighted average of the delay and energy consumption of the UAV u at time t, expressed as: Where ω1 and ω2 are weighting coefficients for time delay and energy consumption, and ω1+ω2=1; The training process of the MADDPG algorithm is shown in Figure 5: at time t With Gaussian noise N t Combined, we get a u (t) is applied to the current vehicle-to-everything (V2X) environment to obtain the reward function r of the drone u. u (t), In order to maximize the reward function and associate it with the objective, the reward function r of the drone u is set. u (t) is: r u (t)=-lnq u (t)#(14) Global information includes state space and action space: state space s i (t) represents all intelligent agents o' u The set of action values ​​(t), where action space A(t) is the action value a of the agent. u The set of (t) is represented as follows: s i (t)={o'1(t),……o' U (t)}#(16) A(t)={a1(t)…a U (t)}#(17) a u (t) Input into the Critic evaluation network, the Critic network calculates the Q value by combining global information (through state sharing among UAVs), where the action value function Q output by UAV u is... u (t) is represented as: in These are parameters of the Critic network, representing the weights of the Critic network, and they do not have fixed specific values. The drone will a u (t) is applied to the current vehicle-to-everything (V2X) environment to obtain the local observation stitched value o' for the next state. u (t+1) and input it into the Actor's target network to obtain the action value a for the next state. u (t+1), and obtain the Q value of the next state through the Critic target network. The calculation formula is as follows: in These are the target Critic network parameters and weights; the Critic network updates its evaluation network parameters by minimizing a loss function. As shown in formula (6): Subsequently, the Actor evaluation network parameters are updated by maximizing the policy objective function. The formula for updating the Actor network policy gradient is: in, Let be the policy objective function, which represents the objective function when the parameter is... Under this strategy, the expected long-term cumulative reward; The formula for calculating the policy gradient is: in, This represents the gradient of the Actor network with respect to its own parameters. Let be the gradient of the Q-value function of the Critic network with respect to the action of agent u; When evaluating parameters in a network and Target network parameters during continuous updates and Update using a soft update method: Where τ is the soft update coefficient, which is a constant between 0 and 1.

Images