Unmanned aerial vehicle group path planning and task offloading method and device for edge computing

Abstract

This invention relates to the field of UAV swarm path planning and task offloading technology, specifically to a method and apparatus for UAV swarm path planning and task offloading oriented towards edge computing. The method constructs a UAV swarm-assisted edge computing system as a two-layer heterogeneous multi-agent decision model, decoupling task offloading and computing resource allocation decisions from flight trajectory control. This allows the higher layer to focus on system-level computing and communication resource optimization, while the lower layer focuses on safe flight and formation maintenance of the UAVs. Structurally, this reduces the strong coupling between flight control, task offloading, and resource allocation, significantly reducing the complexity of the joint optimization problem. Furthermore, by using a decoupled two-layer multi-agent reinforcement learning method to train both the lower and higher layers separately, the stability and convergence reliability of the multi-agent learning process are improved, making the method more practical and deployable in complex dynamic environments.

Description

Technical Field

[0001] This invention relates to the field of UAV swarm path planning and task offloading technology, specifically to a method and apparatus for UAV swarm path planning and task offloading oriented towards edge computing. Background Technology

[0002] UAV-assisted mobile edge computing (UAV-assisted MEC) is considered a crucial technology supporting these applications. It involves deploying computing resources on UAV platforms and leveraging their mobility and flexibility to provide on-demand computing services to ground terminals or other aerial nodes. By strategically planning UAV flight paths and offloading computing tasks to UAV nodes with computing capabilities, task completion latency can be effectively reduced, and the overall system service quality improved.

[0003] However, in real-world complex environments, UAV swarm-assisted MEC systems face multiple challenges. On the one hand, while performing computational services, UAVs also need to complete tasks such as flight control, formation keeping, and obstacle avoidance, resulting in a close coupling between flight trajectory planning and computational task processing. On the other hand, multiple UAVs typically share limited computing and communication resources, and task offloading decisions and computing resource allocation among different nodes influence each other, making the system optimization problem exhibit high-dimensionality, strong nonlinearity, and strong coupling characteristics.

[0004] Existing research often employs centralized optimization, heuristic algorithms, or single-layer reinforcement learning methods to model and solve UAV trajectory planning, task offloading, or resource allocation problems. However, when the scale of UAVs increases or the dynamics of the environment become more pronounced, these methods often face challenges such as high computational complexity, poor scalability, and strong dependence on global information. Furthermore, while multi-agent reinforcement learning methods have certain advantages in distributed decision-making, they are prone to instability during training and difficulty in policy convergence when dealing with heterogeneous decision-making processes such as flight control and task offloading, thus affecting practical application results.

[0005] Therefore, how to coordinate the flight control, task offloading, and computing resource allocation processes of UAV swarms in complex and dynamic environments, reduce the system complexity caused by decision coupling, and improve the stability and practicality of the multi-agent learning process remains a technical problem that needs to be solved in UAV swarm-assisted mobile edge computing. Summary of the Invention

[0006] The purpose of this invention is to provide a method and apparatus for path planning and task unloading of unmanned aerial vehicle (UAV) swarms oriented towards edge computing.

[0007] The technical solution of this invention is as follows:

[0008] A method for path planning and task offloading in a drone swarm for edge computing, used in a drone swarm-assisted edge computing system comprising a leader drone and multiple follower drones, includes the following operations:

[0009] The UAV swarm-assisted edge computing system is constructed as a two-layer heterogeneous multi-agent decision-making model. The upper layer is used for joint decision-making on the task offloading ratio and computing resource allocation, while the lower layer is used for UAV flight trajectory control and formation maintenance.

[0010] At the high level, each follower drone decides its task offloading ratio based on its own mission characteristics, communication conditions, and system load information; the leader drone, based on the offloading ratio of each follower drone, uniformly allocates computing resources to minimize the system task completion delay.

[0011] In the lower layer, each UAV independently outputs continuous flight maneuvers based on local observation information, while simultaneously satisfying UAV dynamics constraints, formation constraints, and obstacle avoidance safety constraints during flight.

[0012] By employing a decoupled two-layer multi-agent reinforcement learning method, the lower and higher layers are trained separately to generate UAV flight trajectories, task offloading decisions, and computational resource allocation strategies, thereby enabling UAV swarm path planning and task offloading.

[0013] In the lower layers: the state space of the leader drone includes its motion state, relative position to the target, and local obstacle information; the action space of the leader drone includes its speed increment and heading increment; and the reward function of the leader drone is constructed from distance advancement reward, target arrival reward, time pressure penalty, first obstacle avoidance and boundary constraint penalty. The state space of the follower drone includes its motion state, relative position information with the leader drone, relative position information with neighboring follower drones, and local obstacle information; and the reward function of the follower drone is constructed from formation distance reward, speed and heading coordination reward, neighbor distance and collision avoidance reward, second obstacle avoidance and boundary constraint penalty.

[0014] At the high level: the state space of the leader drone includes the offloading decisions of each follower drone and the global system load information; the action space of the leader drone includes the computing resource allocation strategy for each follower drone; the reward function of the leader drone is constructed from the system task completion latency, the system-level latency improvement rate, and the offloading computing power mismatch penalty; the state space of the follower drones includes their own task characteristics, communication conditions, and system load information; the action space of the follower drones is the task offloading ratio; the reward function of the follower drones is constructed from the individual task completion latency, the individual-level latency improvement rate, the system-level latency improvement rate, and the bottleneck penalty.

[0015] During the high-level decision-making phase, the follower drones independently decide on the task offloading ratio based on the high-level status, offloading some tasks to the leader drone. Subsequently, the leader drone, based on the offloading decisions of all follower drones, uniformly allocates computing resources.

[0016] After completing high-level decision-making, each UAV executes flight maneuvers based on low-level strategies. The lead UAV updates its flight status according to the mission objectives, and the follower UAV adjusts its speed and heading through local observation to maintain the formation structure.

[0017] A phased training approach is adopted, first training the lower levels and then training the higher levels.

[0018] The low-level training consists of two phases: the first phase trains only the flight control strategy of the leader UAV and updates the network parameters corresponding to the leader UAV; the second phase trains the formation control strategy of the follower UAVs while freezing the low-level network parameters of the leader UAVs and updates only the network parameters corresponding to the follower UAVs.

[0019] An edge computing-oriented UAV swarm path planning and task offloading device, used to implement the aforementioned edge computing-oriented UAV swarm path planning and task offloading method, includes:

[0020] The system modeling module is used to construct the UAV swarm-assisted edge computing system into a two-layer heterogeneous multi-agent decision-making model, in which the high layer is used for task offloading and computing resource allocation decisions, and the low layer is used for UAV flight trajectory control and formation keeping.

[0021] The high-level decision-making module is used to determine the task offloading ratio of each follower drone based on the task characteristics, communication conditions and system load information of the follower drones; and to uniformly allocate the computing resources of the leader drone based on the task offloading ratio to minimize the system task completion delay.

[0022] The low-level flight control module is used to output continuous flight maneuvers based on the local observation information of the UAV; during the execution of flight maneuvers, it simultaneously satisfies the UAV's dynamic constraints, formation constraints, and obstacle avoidance safety constraints.

[0023] The two-layer reinforcement learning processing module is used to train the lower-level flight control module and the higher-level decision-making module respectively using a decoupled two-layer multi-agent reinforcement learning algorithm, so as to output the UAV flight trajectory, mission offloading decision and computing resource allocation strategy.

[0024] A path planning and task offloading device for UAV swarms oriented to edge computing includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the aforementioned UAV swarm path planning and task offloading method for edge computing.

[0025] A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned edge computing-oriented UAV swarm path planning and task offloading method.

[0026] The beneficial effects of this invention are as follows:

[0027] This invention provides a method for UAV swarm path planning and task offloading for edge computing. First, by constructing the UAV swarm-assisted edge computing system as a two-layer heterogeneous multi-agent decision model, task offloading and computing resource allocation decisions are decoupled from flight trajectory control. This allows the higher layer to focus on system-level computing and communication resource optimization, while the lower layer focuses on safe flight and formation maintenance for the UAVs. Structurally, this reduces the strong coupling between flight control, task offloading, and resource allocation, significantly reducing the complexity of the joint optimization problem. Then, at the higher layer, following UAVs independently determine their own task characteristics, communication conditions, and system load information. The decision-making task offloading ratio is determined, and the leader UAV uniformly allocates computing resources from a global perspective to achieve collaborative computing and load balancing among multiple UAVs, thereby effectively reducing the overall system task completion latency. Finally, in the lower layer, each UAV independently outputs continuous flight actions based on local observation information, and simultaneously satisfies dynamic constraints, formation constraints, and obstacle avoidance safety constraints during reinforcement learning training. The lower and higher layers are trained separately using a decoupled two-layer multi-agent reinforcement learning method, which improves the stability and convergence reliability of the multi-agent learning process, making the method of this invention more practical and deployable in complex dynamic environments. Attached Figure Description

[0028] The solutions and advantages of this application will become clear to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0029] In the attached diagram:

[0030] Figure 1 This is a flowchart illustrating the method of this embodiment.

[0031] Figure 2 As shown in the example, this is a comparison of the reward convergence graphs of the leader drone in low-level phased training and joint training.

[0032] Figure 3 As shown in the embodiment, the low-level phased training of the leader UAV training convergence graph;

[0033] Figure 4 In this embodiment, the low-level phased training follows the convergence graph of the UAV training.

[0034] Figure 5In this embodiment, the joint training follows the convergence graph of the UAV training.

[0035] Figure 6 This is a diagram illustrating the impact of changes in the computing resources of the drone on system latency, as shown in the example.

[0036] Figure 7 This is a diagram illustrating the impact of changes in the number of drones on system latency in an example. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the exemplary embodiments of this application clearer, the technical solutions in the exemplary embodiments of this application are described clearly and completely below. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0038] This embodiment provides a drone swarm path planning and task offloading method for edge computing, used in a drone swarm-assisted edge computing system comprising one leader drone and multiple follower drones. See [link to documentation]. Figure 1 This includes the following operations:

[0039] The UAV swarm-assisted edge computing system is constructed as a two-layer heterogeneous multi-agent decision-making model. The upper layer is used for joint decision-making on the task offloading ratio and computing resource allocation, while the lower layer is used for UAV flight trajectory control and formation maintenance.

[0040] At the high level, each follower drone decides its task offloading ratio based on its own mission characteristics, communication conditions, and system load information; the leader drone, based on the offloading ratio of each follower drone, uniformly allocates computing resources to minimize the system task completion delay.

[0041] In the lower layer, each UAV independently outputs continuous flight maneuvers based on local observation information, while simultaneously satisfying UAV dynamics constraints, formation constraints, and obstacle avoidance safety constraints during flight.

[0042] By employing a decoupled two-layer multi-agent reinforcement learning method, the lower and higher layers are trained separately to generate UAV flight trajectories, task offloading decisions, and computational resource allocation strategies, thereby enabling UAV swarm path planning and task offloading.

[0043] The specific steps are detailed below.

[0044] S1. Construct a drone swarm-assisted edge computing system that includes a leader drone and multiple follower drones.

[0045] Specifically, in a 3D disaster relief scenario, consider a drone swarm, with one leader drone. Follow-up drones, assembled A large number of mobile (distributed randomly) people were in the disaster area. U (units) of ground equipment, collectively The equipment does not perform local processing; all data is offloaded to the computing drone. The follower drones first collect data and process it locally, but can optionally offload some data to the leader drone, which has higher computing power. Each follower drone has a set offload ratio. Collection of data types Each follower drone has a different role, collecting only a subset of data. One of them. In this embodiment, the number of following drones must be greater than or equal to the number of data types.

[0046] Each drone is equipped with an edge server, and the computing resources that govern the drone are... Because different types of data require different software, the computing capabilities of drones for heterogeneous data also vary, depending on the data type. The computational efficiency is defined as Follow the drone The computing resources are The drones have a 100% efficiency in processing the collected data types. The drone swarm flies from the starting point to the destination, providing computing services to the ground mobile terminal. At the same time, the drone swarm also needs to avoid obstacles and maintain formation during flight. After the calculation is completed, the results are sent to the ground base station, i.e. the destination. Because this part of the data is small and the transmission time is very short, it can be ignored.

[0047] All drones share a common mission to complete, and the mission cycle for each drone is set to the same duration. , will last Divided into equal parts A uniform time slot, composed of To index, Each time slot The duration is considered short enough to ensure that the drone swarm remains relatively stationary in each time slot, and that the distance to the user remains approximately constant. The drones fly at a fixed altitude.

[0048] In each time slot, the data model representing all users associated with each following drone is as follows: These are the data volume, the number of CPU cycles required to process 1 bit of data, and the data types collected by the drone.

[0049] To simplify the notation, this embodiment uses functional form in the UAV swarm-assisted edge computing system and latency derivation. Represents variables that change over time slots; in decoupled two-layer multi-agent reinforcement learning methods, subscript form is used. Indicates the corresponding variable in the th... The values ​​taken at each time step. The two are numerically equivalent.

[0050] In the UAV swarm-assisted edge computing system of this embodiment, within each time slot, the follower UAVs divide the collected computing tasks into two parts: one part is processed locally, and the other part is offloaded to the leader UAV for computing via an air-to-air communication link. To avoid co-channel interference, all follower UAVs use time-division multiple access (TDMA) during the data transmission phase, sequentially uploading the offloaded task data to the leader UAV. That is, each time slot is divided into a data transmission phase and a computing phase. During the data transmission phase, the follower UAVs offload tasks using TDMA; during the computing phase, the leader UAV processes the received offloaded tasks. Simultaneously, the follower UAVs perform local computing on the remaining un-offloaded task portions. The latency for a follower UAV to complete data processing within a time slot = waiting transmission latency + latency of transmitting the task to the leader UAV + latency of waiting for the leader UAV to process the task + latency of the leader UAV processing the task. The computing process of the leader UAV and the local computing process of the follower UAVs are performed in parallel time.

[0051] Time slot k In this case, the path loss between the follower drone and the leader drone is mainly determined by distance-dependent large-scale propagation attenuation. This embodiment adopts a logarithmic distance path loss model based on free-space propagation, expressed as:

[0052] ,

[0053] For the obstacle occlusion loss term, In time slot k At that time, the leader of the drone and the first i The distance between the following drones Where is the carrier wavelength, and n=2.8 is the path loss exponent, applicable to disaster-stricken environments. It should be noted that the above formula describes the average large-scale path loss caused only by geometric diffusion and propagation distance, while the additional attenuation caused by obstacle occlusion and environmental complexity will be further considered in subsequent models.

[0054] To characterize the additional attenuation of air-to-air (A2A) links caused by obstacles such as buildings and trees in the disaster area environment, an obstacle occlusion loss term is introduced. Unlike existing methods that use LoS / NLoS for probabilistic modeling, this term is used to equivalently describe the additional signal attenuation caused by the presence of obstacles, and its magnitude is dynamically adjusted as the relative position of the UAV changes.

[0055] Specifically, the degree of occlusion of the link in time slot k is defined as follows: This parameter comprehensively reflects the environmental complexity and the impact of obstacles on the propagation path. "Time" indicates an unobstructed situation, while This indicates severe occlusion, and the corresponding obstacle occlusion loss can be modeled as follows: .

[0056] in, This represents the maximum environmentally relevant occlusion loss constant. This linear form is a commonly used equivalent modeling method that maintains computational efficiency for system-level analysis while ensuring physical interpretability. This occlusion loss directly reduces the received signal-to-noise ratio, thus affecting the link's achievable rate and the performance of subsequent task offloading.

[0057] Shadow fading is the random signal attenuation caused by large-scale obstacles. In this model, this embodiment simulates shadow fading by introducing a normal distribution with a standard deviation of 5.0 dB to reflect signal fluctuations caused by occlusion of buildings or other large-scale objects in complex environments. This shadow fading term characterizes the unmodeled environmental randomness and is independent of the aforementioned deterministic occlusion loss term. Shadow fading can be expressed as: , The standard deviation is 5.0 dB, reflecting the random fluctuation amplitude of shadow fading. In the A2A communication scenario, the relative motion between drones will cause channel time changes caused by the Doppler effect. Given that the drone's flight speed is moderate and the carrier frequency is below 6 GHz, the resulting Doppler frequency shift is relatively small and will not have a dominant impact on link performance. Therefore, cross-carrier interference caused by Doppler is not modeled separately.

[0058] In research on unmanned aerial vehicle (UAV) assisted systems, existing methods, to simplify problem analysis, assume that the Doppler effect caused by UAV flight movement can be well estimated, and then compensate for it using advanced algorithms at the receiver. In this embodiment, however, the Doppler effect is treated as an equivalent signal-to-noise ratio (SNR) decrease term to reflect the impact of channel time variations on channel estimation accuracy and adaptive transmission. This modeling method strikes a balance between physical realism and computational efficiency. In this embodiment, a simplified Doppler frequency shift formula is used:

[0059] ,in, For carrier frequency, The component of relative velocity in the communication direction, The speed of light is the maximum speed. A larger frequency shift results in a larger channel estimation error, leading to a decrease in the effective signal-to-noise ratio.

[0060] In summary, during time slot k, the leader drone m and the follower drone... The total path loss between them can be expressed as: , For distance-dependent large-scale path loss This is due to the additional large-scale attenuation caused by obstacle occlusion, and For the shadowing fading term that follows a log-normal distribution, The equivalent additional loss caused by Doppler is used to characterize the channel estimation error caused by time-varying channels.

[0061] Based on the total path loss, the leader drone and the follower drone in the time slot k time Data transmission rate The calculation formula is as follows:

[0062] ,

[0063] in, For system bandwidth, To follow the drone To the launch power of the drone, For noise power spectral density, This represents noise power.

[0064] In drone swarm-assisted edge computing systems, time slots Time i The proportion of follow-up drones will be 100%. The task data is offloaded to the leader drone, and its transmission latency... The calculation formula is:

[0065] ,

[0066] For the first i Total workload of the drones following the flight For time slots k Time i Data transmission rate between the follower drone and the leader drone.

[0067] For the tasks that are not uninstalled, the calculations are performed locally following the drone, and the local calculation latency is... The calculation formula is as follows:

[0068] ,

[0069] Indicates the first i The local computing power of the drone that follows it. For the first i The number of CPU cycles required for a drone to process 1 bit of data.

[0070] For the task of unloading to the leader drone, the leader drone allocates computing resources to each follower drone and processes the task. The latency calculation formula for the leader drone's unloading task is as follows:

[0071] ,

[0072] In time slot k Time assigned to the first The computing resources of the drone that follows it To improve the computational efficiency of the leading drone for the types of tasks collected by each follower drone.

[0073] Within a time slot, the total latency experienced by the drone in completing its mission is determined by both local computation and offloading computation. Since local computation and offloading computation occur in parallel, the total mission completion latency is determined by the larger of the two. The total latency of the unloading task is: This modeling method can accurately depict the cumulative effect of transmission delay under time division multiple access, as well as the delay characteristics brought about by parallel computing in MEC systems.

[0074] In summary, in time slots within, no. The mission completion latency of the drone is determined by the larger of the local computation and the unloading computation. .

[0075] To characterize the overall performance of the entire UAV swarm support system throughout the mission execution cycle, a system-level latency metric is introduced, which measures the system's latency within time slots. The overall latency is defined as the bottleneck latency: The cumulative system latency over the entire task execution cycle is defined as follows: .

[0076] In disaster relief scenarios, missions are highly latency-sensitive. Therefore, this embodiment prioritizes latency performance as the main optimization objective, by jointly optimizing the mission offloading ratio of each follower drone. Leadership in the allocation of drone resources By minimizing system latency using the trajectories of all drones, the optimization problem of the drone swarm-assisted system can be formulated as follows: Minimize instantaneous latency. They are time slots k At that time, the speed of the drone, the first i The speed of the drone following the drone, the course of the drone leading the drone, and the first i The aircraft follows the flight path of the drone.

[0077] In the drone swarm-assisted edge computing system of this embodiment, the following constraints are established: Constraint 1: The sum of the computing resources allocated to the leader drone by all follower drones equals the total computing resources of the leader drone; Constraint 2: The transmit power range of each follower drone and the computing resource constraints of all drones are within their respective threshold ranges; Constraint 3: The distance between the leader drone and follower drones must be within a preset range, i.e., in formation; Constraint 4: The speed of each drone cannot exceed the maximum speed; Constraint 5: The starting and ending positions of the leader drone are the same as the corresponding preset positions; Constraint 6: Follower drones cannot collide with each other, and the distance between them cannot exceed the maximum distance constraint; Constraint 7: The speed and heading increment of each drone within a time slot are within the corresponding threshold range; Constraint 8: The drone motion satisfies dynamics, and the coordinate changes and azimuth angles of each drone satisfy the corresponding preset constraints; Constraint 9: Each drone cannot collide with obstacles (including spheres and cuboids); Constraint 10: The unloading ratio of each follower drone is within the corresponding unloading ratio range.

[0078] S2. Construct the UAV swarm-assisted edge computing system into a two-layer heterogeneous multi-agent decision-making model. The upper layer is used for joint decision-making on the task offloading ratio and computing resource allocation, while the lower layer is used for UAV flight trajectory control and formation maintenance.

[0079] The lower-level multi-agent decision-making model (hereinafter referred to as the lower level) in the two-layer heterogeneous multi-agent decision-making model is used to perform UAV flight trajectory control and formation maintenance. The specific details are as follows.

[0080] In the lower layer, each UAV independently outputs continuous flight maneuvers based on local observation information, while simultaneously satisfying UAV dynamics constraints, formation constraints, and obstacle avoidance safety constraints during flight.

[0081] Specifically, in the low-level flight control phase, the UAV swarm-assisted edge computing system consists of a leader UAV and multiple follower UAVs, forming a heterogeneous multi-agent decision-making system with a leader-follower structure. Different types of UAVs perform different functions in the system, and their state space, action space, and reward function are different.

[0082] Within each time step, each UAV independently decides its flight actions based on its own observation status and learns its strategy through decentralized training and decentralized execution.

[0083] In this embodiment, the state components of all lower-level agents are normalized before being input into the algorithm network to improve training stability.

[0084] For a leader drone, its state space, action space, and reward function are defined as follows.

[0085] The state space of the drone includes its motion state, relative target position, and information about local obstacles.

[0086] Specifically, time slots k At that time, the state space of the leading drone ,in, For time slots k The position of the leadership drone in the two-dimensional plane is the current position of the leadership drone. They are time slots k The system displays the speed and heading angle of the drone in the current time slot, facilitating speed and heading coordination with other drones. The target point is defined as a two-dimensional location. The drone being observed is its current position relative to the target point. For time slots k The location of local obstacles as observed by the drone during the leadership period.

[0087] The action space of a leader drone is defined as a continuous action space, including velocity increments, heading increments, and time slots. At time, action space , They are time slots k The speed and heading increments of the drone are controlled by the time. This incremental motion design effectively limits the drone's maneuverability, ensuring a smooth trajectory that conforms to actual flight constraints.

[0088] The reward function for the leader drone is designed to guide it to reach the target area within a limited time while meeting safety constraints. It is constructed from distance advancement rewards, target arrival rewards, time pressure penalties, first obstacle avoidance and boundary constraint penalties.

[0089] The reward function for the leader drone is as follows:

[0090] ,

[0091] , , , , These are the reward function value for the leading drone, the distance advancement reward, the target arrival reward, the time pressure penalty, and the first obstacle avoidance and boundary constraint penalty.

[0092] Distance Advancement Rewards The formula for encouraging drones to continuously fly towards the target area is as follows:

[0093] ,

[0094] Indicates time slot kThe Euclidean distance between the drone and the target point at that time. Indicates time slot k The target distance at -1 is used to measure the advance in adjacent time steps. For time slots k The current location of the drone under the leadership of the leader. The target point location, As a distance advancement reward weighting coefficient, this encourages leading drones to shorten the distance to the target in each time step.

[0095] Reward for reaching the target Significant rewards are given for successful drone entry into the target area, calculated using the following formula:

[0096] ,

[0097] Indicates the radius of the target area. To reach the target reward weight, This is an indicator function, and this term is a sparse reward used to guide the drone to successfully reach the target area, which is the core mission objective.

[0098] Considering the need for emergency response missions in disaster areas within a limited total mission time Completed within the allotted time, a time pressure penalty is introduced when the flight duration approaches the designated mission time. The calculation formula is as follows:

[0099] ,

[0100] The maximum number of permissible flight slots, This indicates the threshold time at which time pressure begins to take effect. As a time pressure penalty weight, this item is used to prevent the leader drone from using inefficient flight strategies when the mission deadline is approaching. This time pressure penalty is a soft constraint that does not directly optimize flight time, but guides the leader drone to complete the mission within a limited time.

[0101] To ensure flight safety, obstacle avoidance and boundary constraint penalties are introduced as safety penalties. The first obstacle avoidance and boundary constraint penalty... The calculation formula is as follows:

[0102] ,

[0103] , These represent the distances between the leading drone and the nearest obstacle and the environmental boundary, respectively. , These are the nearest obstacle penalty weight and the environmental boundary penalty weight for the leading drone, respectively. , These are the first and second penalty functions, which are monotonically increasing with respect to distance, used to prevent collisions with obstacles and boundary violations.

[0104] At the lower level, for a drone follower, its state space, action space, and reward function are defined as follows.

[0105] Time slot k At that time, the first i State space of the drone This includes the current motion status, relative position information with the leader drone, relative position information with neighboring follower drones, and local obstacle information, as shown in the following formula:

[0106] ,

[0107] For time slots k Time i The system follows the drone's position in a two-dimensional plane, i.e., follows the drone's current position. They are time slots k Time i The aircraft follows the speed and heading angle of the drone in the current time slot. The first i The relative position, relative speed, and relative heading of the follower drone and the leader drone. , , They represent time slots respectively. k The position, speed, and heading angle of the drone were monitored at that time. Indicates time slot k Time i The relative position information of the drone following another drone. To track the location of local obstacles observed by the drone.

[0108] At the lower level, all follower drones and the leader drone have the same motion space; the drone's motion space... , , They are time slots k Time i The speed and heading increments of the drone are followed.

[0109] In the lower-level flight control phase, the reward function for the following drone aims to guide it to stably follow the leader drone and maintain a reasonable formation structure while meeting safety constraints. To this end, time slots... k At that time, the first iThe reward function for the following drone is constructed from the formation distance reward, the speed and heading cooperative reward, the neighbor distance and collision avoidance reward, and the second obstacle avoidance and boundary constraint penalty. The calculation formula is as follows:

[0110] ,

[0111] , , , , These are the reward function values ​​for following the drone, formation distance reward, speed and heading coordination reward, neighbor distance and collision avoidance reward, and second obstacle avoidance and boundary constraint penalty, respectively.

[0112] To guide the follower drone and the leader drone in maintaining a stable communication and cooperative flight relationship, this embodiment does not impose a rigid constraint on a single precise formation distance. Instead, it introduces a desired formation distance range and designs a continuous reward function based on the distance deviation, with time slots... k Time i Formation distance reward for following drones The calculation formula is as follows:

[0113] ,

[0114] Indicates time slot k Time The Euclidean distance between the follower drone and the leader drone. To determine the desired formation distance from the center, The standard deviation of the formation distance is used to measure the tolerance width of the formation distance bonus. As the formation distance reward weight, this reward function in Falling into It achieves higher values ​​when the distance is within the expected formation distance range centered on the target distance, and smoothly decays when the distance deviates from this range, thereby improving the robustness of formation control under perception error and discrete control conditions.

[0115] To enhance the stability of formation flight, a combined speed and heading reward is introduced to encourage follower drones to maintain consistent motion with the lead drone. This embodiment includes time slots. k At that time, the first i Speed ​​and heading coordination rewards for following drones The calculation formula is as follows:

[0116] ,

[0117] ,

[0118] , They represent time slots respectively. k Time i The speed and size of the follower drone and the leader drone Indicates time slot k Time i The heading angle difference between the follower drone and the leader drone. , They are time slots k Time i The heading angle of the drone that follows and leads the drone. , These represent the speed coordination reward weight and the heading coordination reward weight, respectively. This reward item encourages the following drones to maintain consistency with the leader drone in speed and heading, thereby reducing formation oscillation.

[0119] To avoid excessive clustering or collisions among following drones while maintaining a reasonable spatial distribution structure, this embodiment introduces a collision avoidance reward based on the neighbor distance within the desired neighbor spacing range. The calculation formula is as follows:

[0120] ,

[0121] ,

[0122] For time slots k Time setting number The average neighbor distance between a follower drone and other follower drones. The desired center value of neighbor spacing. The standard deviation of neighbor distances is used to measure the tolerance width of the expected range of neighbor distances. Indicates the total number of following drones. For the first The position vector of the aircraft is followed. For the first The position vector of the aircraft is followed. , The minimum safe distance threshold, The reward function provides a positive incentive when the average distance between neighbors falls near the desired separation range, thereby encouraging the drone swarm to maintain a uniform distribution; when the neighbor distance is less than a safety threshold... At the same time, explicit collision penalties are introduced to strengthen safety constraints and prevent dangerous close encounters or collisions between drones.

[0123] Time slot k Time iThe calculation formula for the obstacle and boundary constraint penalty term for the drone following the obstacle avoidance is as follows:

[0124] ,

[0125] in, , , respectively, are the distances between the i-th following drone and the nearest obstacle and the environmental boundary at time slot k. , It is a monotonically increasing penalty function with respect to distance, used to prevent collisions with obstacles and out-of-bounds behavior. , These are the nearest obstacle penalty weight and the environmental boundary penalty weight for the i-th following drone, respectively.

[0126] At the high level, each follower drone decides its task offloading ratio based on its own mission characteristics, communication conditions, and system load information; the leader drone, based on the offloading ratio of each follower drone, uniformly allocates computing resources to minimize the system task completion delay.

[0127] Specifically, during the high-level task offloading and resource allocation phase, the system also consists of a leader drone and multiple follower drones, forming a heterogeneous multi-agent decision-making system with a leader-follower structure. Due to the differences in computing power and system responsibilities among different types of drones, their state space, action space, and reward function are inconsistent.

[0128] Therefore, the high-level task offloading and resource allocation problem is modeled as a heterogeneous multi-agent Markov decision process. Within each decision cycle, the following UAV makes task offloading decisions based on its own task status and system environment information; the leader UAV, based on the global system state, coordinates and allocates computational resources for the received tasks to minimize the overall system task completion latency.

[0129] In the high-level context, the state space, action space, and reward function of the leadership drone are defined as follows.

[0130] The state space of the leader drone includes the unloading decisions of each follower drone and the system's global load information. , This is the global system sub-state vector. This represents the current timeline. To lead the overall computing power of drones, To unload the total amount of data, In time slot k No. i Total workload of the drones following the flight In time slot k No.i The unloading ratio of the drone following the aircraft The total number of CPU cycles required to unload data. In time slot k Processing the i The number of CPU cycles required for a drone to follow. For time slots k Time i The sub-state vector of the current time slot mission information of the drone is followed. They are time slots k Time i The total workload of the drone, the number of CPU cycles required to process 1 bit of data, and the types of data collected. For time slots k Time i The transmission rate of the follower drone and the leader drone. For time slots k Time i The time required for all local processing by the drone. They are time slots k Time i The computational efficiency of the task types collected by the follower drones and the offloading ratio of data transferred to the leader drone are considered. Through this state design, the leader drone can comprehensively consider the task characteristics and system load of each follower drone from a global perspective, thereby achieving efficient scheduling of computational resources.

[0131] The action space of the leader drone includes the computational resource allocation strategy for each follower drone, defining the action space. , For time slots k Time assigned to the first The computing resource share of the drone following the drone, and meets the total computing power of the leading drone. constraint: .

[0132] The reward function for the leader drone is constructed from the system task completion latency, system-level latency improvement rate, and offloading computing power mismatch penalty term. The calculation formula is as follows:

[0133] ,

[0134] ,

[0135] ,

[0136] For system task completion delay, For system-level latency improvement rate, , These represent the reference system delay value and the system delay at time slot k, respectively. The first parameter is denoted as , and the minimum value is denoted as . To unload the computing power mismatch penalty item, , , These are weights for system task completion latency, system-level latency improvement rate, and offloading computing power mismatch penalty. When a follower drone chooses to offload a task, but its task completion latency after offloading is significantly worse than the latency of fully local computation under the current computing power allocation, In time slot k Time i The unloading ratio of the drone following the aircraft For time slots k Time i The mission completion delay of the drone. For time slots k Time i The time required for all local processing by the drone. A threshold for determining whether a significant unloading has occurred. This is a tolerance factor used to account for communication overhead. This penalty is activated only when the unloading behavior of the follower drone is clearly unreasonable, and is used to constrain the leader drone from blindly accepting unloading tasks under resource constraints.

[0137] The aforementioned reward function for the high-level leadership drone prioritizes latency minimization, avoiding the introduction of excessive auxiliary reward items to reduce the difficulty of policy learning. The separation of system-level and individual-level rewards allows the leadership drone to focus on global resource scheduling, while follower drones concentrate on individual offloading decisions. Lightweight collaborative items also maintain overall system performance.

[0138] In the high-level context, the state space, action space, and reward function for a follow-up drone are defined as follows.

[0139] In high-level task offloading decisions, the state space of the following drone not only includes its own task characteristics and communication conditions, but also explicitly encodes system load-related information, such as expected transmission latency and current computational load factors. This design enables the following drone to perceive the resource congestion level of the leader drone without relying on additional queuing penalties, thereby achieving load-oriented offloading decisions and implicit load balancing, which helps improve the decision stability and scalability in multi-drone collaborative computing scenarios.

[0140] The state space of the drone includes its own mission characteristics, communication conditions, system load information, and time slots. k Time iState space of the drone ,in, Represents the task feature sub-state vector. They are time slots k Time i The total workload of the drone, the number of CPU cycles required to process 1 bit of data, and the types of data collected. For time slots k Time i The sub-state vector represents the local processing capability of the drone following it, i.e., the time during which all data is processed locally by the drone following it. This is the communication and unloading conditional sub-state vector. For time slots k Time i The communication transmission rate between the follower drone and the leader drone For time slots k Time i The expected transmission latency of the drone following the aircraft. To lead the development of drones for the first i The computational efficiency of the task types collected by the drone. This represents the system load-aware sub-state vector. They are time slots k -1 Hour Command Drone Load Information, Time Slot k The current overall load of the time slot system. For time slots k Under ideal and fair allocation conditions, the share of computing resources that a single follower drone can obtain is... For time slots k -1 hour i The unloading ratio of the follower drone. Through the above state design, the follower drone can simultaneously perceive its own mission urgency, communication costs, and system-level resource competition when deciding on the unloading ratio.

[0141] Time slot k Time i The motion space of the drone following the mission is the proportional motion space for mission unloading. , This indicates that the task is processed entirely locally. This indicates that the task is completely unloaded to the leader drone, and the remaining values ​​correspond to the proportion of unloading.

[0142] Time slot k Time i The reward function for the drone is constructed from the individual task completion latency, the individual-level latency improvement rate, the system-level latency improvement rate, and the bottleneck penalty. The formula for calculating the reward function is as follows:

[0143] ,

[0144] ,

[0145] ,

[0146] , , , , These represent the reward function value of the i-th following UAV at time slot k, the individual task completion latency, the individual-level latency improvement rate, the system-level latency improvement rate, and the bottleneck penalty, respectively. For time slots k Time i The mission completion delay of the drone. For time slots k Time i The time required for all local processing by the drone. The first parameter is the minimum value, used to avoid numerical instability. The system delay is at time slot k. This is a tolerance coefficient used to suppress the excessive drag on the overall system performance caused by individual decisions. , , , These are, respectively, the individual task completion latency weight, the individual-level latency improvement rate weight, the individual-level latency improvement rate weight, and the bottleneck penalty weight. Encourage minimizing the delay in completing your own tasks. Rewards are given for individual improvements compared to fully local processing. Introducing system-level collaborative signals helps prevent overly selfish individual decisions. This is a mild bottleneck penalty.

[0147] It should be noted that the latency improvement rate is calculated using the completion latency of "tasks processed entirely locally" as a unified benchmark, rather than the local computation latency under the current offloading decision. This benchmark based on fully local processing provides a consistent comparison scale for different offloading ratios, thereby avoiding the reward function from biasing the offloading ratio itself and more accurately characterizing the actual benefits of offloading and resource allocation strategies relative to the no-offloading solution.

[0148] S3. A decoupled two-layer multi-agent reinforcement learning method is used to train the lower and higher layers separately to generate UAV flight trajectories, task offloading decisions, and computational resource allocation strategies, thereby executing UAV swarm path planning and task offloading.

[0149] In the decoupled two-layer multi-agent reinforcement learning method, in time slots In this case, the policy optimization objective of the decoupled two-layer multi-agent reinforcement learning method can be defined by introducing a policy entropy regularization term based on the expected cumulative reward, as shown in the following formula:

[0150] ,

[0151] for, Indicates the policy in the state The entropy below, The temperature coefficient is used to balance performance optimization and exploration capability. By introducing an entropy regularization term, the policy can maintain a moderate degree of randomness during the learning process, thereby avoiding premature convergence to a suboptimal solution.

[0152] To evaluate the performance of the policy in a continuous action space, a decoupled two-layer multi-agent reinforcement learning method simultaneously learns the state-action value function. This value function minimizes the mean squared error loss function. To update, define as:

[0153] ,

[0154] Indicates the target network parameters. , , , In time slots k The state, the actions taken, and the time slot The state and actions taken for The minimum value of the two target value networks is taken during time slots to prevent overestimation and improve the stability of the training process. , They are time slots k Instant rewards, discount factors.

[0155] Based on this, the policy network is updated by minimizing the following objective function:

[0156] ,

[0157] Action samples generated based on reparameterization techniques, Using temperature as a coefficient, this update method allows the strategy to gradually converge toward areas of high-value actions while maintaining its exploratory capabilities.

[0158] To avoid manually adjusting the entropy weights, this embodiment employs an adaptive temperature parameter mechanism, minimizing the entropy target deviation. The system is updated to dynamically balance strategy exploration and performance optimization; the loss function is:

[0159] ,

[0160] Let be the target entropy.

[0161] In solving the two-layer heterogeneous multi-agent decision-making model of a UAV swarm-assisted edge computing system based on a decoupled two-layer multi-agent reinforcement learning method, the input includes the UAV swarm-assisted collaborative processing task environment and a discount factor. Maximum number of training rounds for low-level flight control Maximum number of training rounds for high-level task unloading Experience replay pool batch size Target entropy of each agent Output the high-level offloading and resource allocation strategy after training is completed. The low-level flight and formation control strategies that have been trained .

[0162] In addition, during the training process, a phased training method is adopted, which first trains the lower levels and then trains the higher levels. The specific details are as follows.

[0163] During the training of low-level flight and formation control strategies, a maximum number of time slots is set for each episode. Each time slot acquires the initial low-level state. Low-level training consists of two phases. The first phase trains only the flight control strategy of the leader drone and updates its corresponding network parameters. That is, it focuses on training the leader drone to safely reach the target point, updating only the leader drone's network. After the leader drone can stably complete the task, the low-level leader drone network parameters are frozen, and the training focuses on the follower drones, updating only their networks. The second phase, with the leader drone's low-level network parameters frozen, trains the formation control strategy for the follower drones, updating only their corresponding network parameters.

[0164] For each time slot Each drone operates according to its current low-level strategy. Select flight control actions , Perform flight maneuvers and update position status, calculate the leader drone flight reward. Rewards for following drone formations And obtain the next state. , . Empirical tuples Store in the corresponding experience replay pool , If the replay pool data meets the update conditions, then from Random sampling A sample of experiences The update method is based on a decoupled two-layer multi-agent reinforcement learning approach: updating the Critic network parameters, updating the Actor network parameters (selecting the update target based on the current training stage), and automatically adjusting the temperature parameters. After the low-level training is completed, all low-level policy network parameters are frozen for use in high-level training.

[0165] During the training of high-level task offloading and resource allocation strategies, the environment is reset, and each time-slot agent acquires the initial high-level state. , For each time slot Follow the drone according to and strategies Offloading ratio of decision-making tasks After sensing all unloading decisions, the leader drone, based on the strategy... Allocate computing resources to unload tasks (i.e., action) Invoke the frozen low-level strategy; Perform specific flight control and formation maintenance; calculate mission completion time delay indicators and obtain high-level rewards. , And observe the state of the next high level. , , experience tuple Store in high-level experience replay pool If the replay pool data meets the update conditions, then from Random sampling A sample of experiences Update the centralized Critic network parameters, update the high-level Actor network parameters for each UAV, and automatically adjust the temperature parameters. .

[0166] This embodiment employs a discrete time-slot modeling approach, where each time slot corresponds to a single task offloading decision and flight trajectory update process within the environment. Within each time slot, the system sequentially completes high-level task offloading and resource allocation decisions, low-level flight control execution, and reward feedback and status updates, forming a complete closed-loop decision-making process.

[0167] The above-mentioned decoupled two-layer multi-agent reinforcement learning method is used to solve and construct a two-layer heterogeneous multi-agent decision model, and the specific steps for obtaining the UAV flight trajectory, task offloading decision and computing resource allocation strategy are as follows.

[0168] Step 1, State Observation. In the time slot... Initially, each agent constructs observation information based on the current system state. The follower drone first builds a high-level state based on its own mission characteristics, communication conditions, and system load information for subsequent offloading decisions. The low-level state consists of each drone's position information, speed, target point location, and relative position information with neighboring obstacles and follower drones, used for flight control. The leader drone's high-level state is generated after the follower drones complete their offloading decisions, to perceive the current system's offloading load and computational demand distribution.

[0169] Step 2: High-Level Task Offloading and Resource Allocation Decisions. During the high-level decision-making phase, follower drones independently decide on the task offloading ratio based on their high-level status, offloading some tasks to the leader drone. Subsequently, based on the offloading decisions of all follower drones, the leader drone uniformly allocates its computing resources to each offloaded task to meet its total computing power constraints. This high-level decision-making process embodies a "first offload, then allocate" sequential structure, which helps the leader drone coordinate resources from a global perspective.

[0170] Step 3: Low-level Flight and Formation Control. After completing high-level decisions, each UAV executes flight maneuvers based on low-level strategies. The lead UAV updates its flight status according to the mission objectives, while follower UAVs adjust their speed and heading through local observations to maintain formation structure. Throughout this process, all UAVs must ensure flight safety. During execution, the UAV swarm system constrains minimum safe distances to prevent excessive proximity or collisions between UAVs.

[0171] Step 4: Action Execution and Reward Calculation. After the action is executed, the system calculates the task completion latency and system bottleneck latency for each UAV within that time slot based on the current unloading ratio, resource allocation results, and flight status. High-level rewards focus on task completion latency and its improvement relative to the all-local processing baseline, while low-level rewards primarily reflect flight stability, formation keeping, and safety.

[0172] Step 5: Status Update and Time Step Update. The drone swarm system updates the drone positions, mission progress, and statistics, and then proceeds to the next time slot. When the termination condition is met (i.e., the mission reaches the specified total time or any drone collides with the target drone), the current round ends; otherwise, the decision-making process for the next time slot continues.

[0173] Through the above process, this embodiment achieves coordinated optimization of high-level task offloading and low-level flight control, enabling multi-UAV systems to complete efficient and stable collaborative computing tasks in a dynamic communication and computing environment.

[0174] To verify the effectiveness of the method in this embodiment, a simulation experiment was conducted.

[0175] Figure 2 A comparison of the leader drone reward convergence graphs is presented between low-level phased training (training the leader drone first, then the follower drone) and existing joint training methods (training both the leader and follower drones together). Under the same training rounds, phased training converges faster and yields higher rewards because the leader drone focuses solely on navigation and obstacle avoidance, unaffected by the follower drone colliding with obstacles and causing premature round termination, to reach the target point. In terms of the leader drone's success rate in reaching the target point, phased training outperforms joint training on average. Furthermore, the slight oscillations in the curves indicate that the entropy regularization of the decoupled two-layer multi-agent reinforcement learning method in this embodiment (the initial entropy gradually approaches the leader drone's target entropy during training) continues to explore throughout the training process. This exploration prevents premature convergence to suboptimal policies and enhances the leader drone's robustness to dynamic environmental changes.

[0176] Figure 3 , Figure 4 The images show the convergence graphs for the low-level, phased training of the leader drone and the follower drones, respectively. After the leader drone pre-converges, the follower drones are trained. The reward curve of the follower drones converges more smoothly because the leader drone they follow can stably reach the target point. The designed reward function can continuously guide the process, and the training time is longer than that of the leader drone. The continuous small oscillations reflect the adaptive exploration mechanism of the decoupled two-layer multi-agent reinforcement learning method in this embodiment. The smooth and moderately oscillating reward curve indicates stable convergence without overfitting. In terms of formation success rate, phased training is superior to joint training.

[0177] Figure 5 The training convergence graph for the joint training drone shows significant periodic oscillations. Although the rewards fluctuate within a certain range in the later stages of training, the large differences in rewards lead to policy instability. This instability is due to the unstable strategy of the leader drone, and any drone colliding with an obstacle will prematurely terminate the round.

[0178] Furthermore, compared to the smooth convergence curve of the score-phase training, changes in entropy also have a significant impact on the curve oscillations:

[0179] In phased training (see...) Figure 4 Since the leader drone's strategy has converged, and the follower drone faces a stable learning environment, the entropy value continuously increases during training. In the later stages of training, the entropy value approaches the target entropy, indicating good convergence. This trend reflects the shift in the decoupled two-layer multi-agent reinforcement learning method of this embodiment from initial exploration (high randomness) to later utilization (deterministic strategy). The strategy tends towards deterministic behavior in the later stages, but still retains some exploratory ability to cope with dynamic environments. The reward curve shows a smooth upward trend with very small oscillations. In contrast, joint training (see...) Figure 5The entropy value in the model is extremely unstable and rises rapidly. This continuous increase stems from the non-stationarity of the leader drone's strategy: whenever the follower drone learns to adapt to the current leader drone's strategy, the leader strategy changes immediately, the learned behavior becomes invalid, and the reward decreases. In this embodiment, the adaptive entropy mechanism of the decoupled two-layer multi-agent reinforcement learning method misjudges that the exploration is still insufficient, further increasing the entropy value to encourage randomization. Therefore, the reward curve oscillates violently, and the strategy is unstable.

[0180] Figure 6 To understand the impact of changes in UAV computing resources on system latency, we can...

[0181] Local computation latency is a flat line, unaffected by changes in the leader drone's computing resources. This strategy completely avoids communication overhead and serial upload penalties, but limited by the fixed local computing power of the follower drones, it cannot utilize the high-performance resources of the leader drone. Therefore, when the leader drone has sufficient computing resources, the latency is worse than random offloading. Full offloading latency continuously decreases as the leader drone's computing resources increase, but it remains at the highest level. Meanwhile, in low-computation-resource areas, this strategy performs worst, mainly because the leader drone's post-allocation processing capability is weaker than local processing, resulting in significantly higher computation latency than local processing. Furthermore, TDMA serial upload further amplifies transmission overhead. Even in high-computation-resource areas, full offloading cannot utilize the follower drones' local computing power, therefore the latency reduction is limited and remains higher than other strategies. Random offloading partially utilizes the local computing resources of the follower drones while reducing the amount of data uploaded serially, thus avoiding the extreme communication and queuing penalties of full offloading. As the computing resources of the leader drone continue to increase in the later stages, random offloading gradually becomes superior to local processing. This is because the computing resources allocated by the leader drone to each follower drone increase, and the performance of local processing is inevitably better than local processing when the leader drone is processing tasks simultaneously.

[0182] The decoupled two-layer multi-agent reinforcement learning method in this embodiment (corresponding to the MASAC algorithm in the figure) consistently achieves the lowest latency across the entire computing resource range of the leader drone, demonstrating significant superiority and adaptability. In low-computational-resource areas, this decoupled two-layer multi-agent reinforcement learning method learns a strategy of prioritizing local computation with appropriate offloading, effectively alleviating the computing bottleneck of the leader drone while avoiding queuing penalties caused by excessive uploading. As the computing resources of the leader drone increase, this decoupled two-layer multi-agent reinforcement learning method adaptively increases the offloading ratio, fully utilizing high-performance resources to significantly reduce latency. In high-computational-resource areas, the offloading ratio of this decoupled two-layer multi-agent reinforcement learning method continues to increase, shifting the system bottleneck to the communication side. Therefore, the latency reduction is slightly slower when computing resources are increased later, but overall, the performance is optimal, and increasing computing resources is significantly better than local computation, demonstrating the advantages of edge computing.

[0183] Figure 7 To understand the impact of changes in the number of follower drones on system latency, we can see that when all tasks are offloaded, latency exhibits a significant near-linear growth trend with the increase in the number of follower drones. This strategy forces all tasks to be offloaded to the leader drone, causing a sharp amplification of the queuing effect under the time-division multiple access (TDMA) serial upload mechanism: adding a follower drone not only increases its own transmission overhead but also increases queuing latency (i.e., serial upload latency), further exacerbating the communication bottleneck. Simultaneously, the randomness of task data volume and computational requirements introduces fluctuations, resulting in an unbalanced processing load on the leader drone, ultimately leading to the highest overall latency. When computing locally, latency shows a gradual upward trend with the increase in the number of follower drones. This strategy relies entirely on the local computing power of the follower drones and the number of CPU cycles required to process tasks, avoiding communication overhead and serial queuing penalties. Therefore, the latency of the follower drone itself is mainly affected by the heaviest task, without interference from other follower drones. However, the system latency is affected; for each additional follower drone, the maximum latency is taken as the system latency, hence the slight fluctuations in the curve, stemming from the random distribution of task volume and computational density. While this strategy is stable, it cannot utilize the high-performance resources of the leader drone and gradually becomes a performance lower bound as the scale expands. During random unloading, latency increases significantly with the number of follower drones, with a growth rate between that of full unloading and local computation. This strategy lacks intelligent decision-making, leading to unnecessary unloading of some tasks and underutilization of resources, resulting in overall inefficiency. Its volatility further reflects the instability of random decision-making in strongly coupled environments.

[0184] The decoupled two-layer multi-agent reinforcement learning method in this embodiment achieves the lowest latency across all follower drone numbers, demonstrating significant superiority and scalability. In small-scale scenarios with a small number of follower drones, this method significantly outperforms the local computational baseline through adaptive offloading and resource allocation, fully leveraging the high performance of the leader drone while avoiding queuing penalties caused by excessive offloading. While latency increases slightly with the number of follower drones, the increase is still far lower than other strategies and remains optimal. In large-scale scenarios, the latency of this method gradually approaches the local computational baseline. This is mainly due to the reduced allocable resources per follower drone caused by the fixed leader drone's computational resources and the increased serial upload penalty, leading the algorithm to adaptively reduce the offloading ratio and favor a local-first strategy. Nevertheless, this method maintains a significant advantage through adaptive coordination decision-making, demonstrating its robustness and adaptability in non-stationary, multi-agent coupled environments.

[0185] This embodiment also provides a drone swarm path planning and task offloading device for edge computing, used to implement the above-mentioned drone swarm path planning and task offloading method for edge computing, including:

[0186] The system modeling module is used to construct the UAV swarm-assisted edge computing system into a two-layer heterogeneous multi-agent decision-making model, in which the high layer is used for task offloading and computing resource allocation decisions, and the low layer is used for UAV flight trajectory control and formation keeping.

[0187] The high-level decision-making module is used to determine the task offloading ratio of each follower drone based on the task characteristics, communication conditions and system load information of the follower drones; and to uniformly allocate the computing resources of the leader drone based on the task offloading ratio to minimize the system task completion delay.

[0188] The low-level flight control module is used to output continuous flight maneuvers based on the local observation information of the UAV; during the execution of flight maneuvers, it simultaneously satisfies the UAV's dynamic constraints, formation constraints, and obstacle avoidance safety constraints.

[0189] The two-layer reinforcement learning processing module is used to train the lower-level flight control module and the higher-level decision-making module respectively using a decoupled two-layer multi-agent reinforcement learning algorithm, so as to output the UAV flight trajectory, mission offloading decision and computing resource allocation strategy.

[0190] This embodiment also provides a drone swarm path planning and task offloading device for edge computing, including a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the above-described drone swarm path planning and task offloading method for edge computing.

[0191] This embodiment also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-described edge computing-oriented UAV swarm path planning and task offloading method.

[0192] This embodiment provides a method for UAV swarm path planning and task offloading for edge computing. First, by constructing the UAV swarm-assisted edge computing system as a two-layer heterogeneous multi-agent decision model, task offloading and computing resource allocation decisions are decoupled from flight trajectory control. This allows the higher layer to focus on system-level computing and communication resource optimization, while the lower layer focuses on safe flight and formation maintenance of the UAVs. Structurally, this reduces the strong coupling between flight control, task offloading, and resource allocation, significantly reducing the complexity of the joint optimization problem. Then, at the higher layer, following UAVs independently determine their own task characteristics, communication conditions, and system load information. The decision-making task offloading ratio is determined, and the leader UAV uniformly allocates computing resources from a global perspective to achieve collaborative computing and load balancing among multiple UAVs, thereby effectively reducing the overall system task completion latency. Finally, in the lower layer, each UAV independently outputs continuous flight actions based on local observation information, and simultaneously satisfies dynamic constraints, formation constraints, and obstacle avoidance safety constraints during reinforcement learning training. The lower and higher layers are trained separately using a decoupled two-layer multi-agent reinforcement learning method, which improves the stability and convergence reliability of the multi-agent learning process, making the method of this invention more practical and deployable in complex dynamic environments.

[0193] This embodiment provides a path planning and task offloading method for UAV swarms oriented towards edge computing. At the lower-level flight trajectory control layer, it ensures the UAV swarm safely reaches the target point and maintains formation. At the higher-level decision-making level, it directly outputs the continuous computing resource allocation ratio, achieving integrated optimization from task offloading to computing resource allocation. This sequential decision-making structure of "offloading first, then allocating" effectively reduces the complexity of the high-dimensional joint action space.

[0194] This embodiment provides a path planning and task offloading method for UAV swarms oriented towards edge computing. It designs a hybrid training and execution architecture for two-layer heterogeneous decision-making, avoiding simplistic categorization of the system into traditional "fully CTDE" or "fully decentralized" paradigms. The high-level layer (task offloading and resource allocation layer) employs centralized value evaluation: during the training phase, a centralized commentator uses joint state-joint action as input to stably evaluate system-level latency benefits; during the execution phase, follower UAVs output offloading ratios based solely on their local high-level observations, while the lead UAV allocates computing power based on received offloading requests, thus balancing global coordination and system latency optimization. The low-level layer (safe flight and formation layer) employs decentralized execution and heterogeneous value learning: each UAV independently outputs continuous flight actions based on local observations; simultaneously, to improve sample efficiency and scalability, low-level follower UAVs share a policy network and a shared experience replay pool, but retain their own independent value networks to achieve differentiated updates under individual rewards, thereby mitigating non-stationarity and gradient noise issues in multi-agent training.

[0195] This embodiment provides a path planning and task offloading method for UAV swarms oriented towards edge computing. Considering that UAV swarm assistance faces factors such as dense obstacles and strict formation spacing constraints in real urban scenarios, this embodiment introduces a safety layer constraint and safety correction mechanism in the low-level flight trajectory control to improve the executability and training stability of the strategy.

[0196] This embodiment provides a drone swarm path planning and task offloading method for edge computing. It constructs a neighbor distance and collision avoidance reward, guides the drone swarm to maintain a reasonable spatial distribution through the "expected neighbor distance", and triggers explicit penalties when the average neighbor distance is lower than the minimum safety threshold, thereby significantly reducing the probability of excessive drone aggregation and close-range collisions.

[0197] This embodiment provides a drone swarm path planning and task offloading method for edge computing. After a drone performs an action, an online correction step of minimum safe distance is introduced to correct potentially unsafe motion updates. The correction cost is fed back as a penalty to the reward of lower-level following drones. This mechanism reduces the drastic perturbation caused by collision samples during training, facilitating formation learning and improving the stability and convergence reliability of the training process.

[0198] This embodiment provides a path planning and task offloading method for UAV swarms oriented towards edge computing. It sets independent target entropies for each leader UAV and each follower UAV, and adjusts adaptive temperature parameters to maintain a balance between exploration intensity and policy convergence speed during training, thereby improving the robustness of multi-agent collaborative learning. Combined with a two-layer training strategy (training lower-layer flight control first, then training higher-layer offloading and resource allocation), the lower layer adopts a phased learning process of "leader UAV training first, follower UAV training later," and freezes the converged leader UAV policy network after phase switching, reducing the non-stationarity of joint learning. This design avoids the problem of unbalanced exploration intensity among multiple agents and significantly improves overall convergence stability.

[0199] While exemplary embodiments of the invention have been described herein, many other variations or modifications conforming to the principles of the invention can be directly determined or derived from the disclosure of the invention without departing from its spirit and scope. Therefore, the scope of the invention should be understood and recognized to cover all such other variations or modifications.

Claims

1. A method for path planning and task offloading in a drone swarm for edge computing, used in a drone swarm-assisted edge computing system comprising a leader drone and multiple follower drones, characterized in that, This includes the following operations: The UAV swarm-assisted edge computing system is constructed as a two-layer heterogeneous multi-agent decision-making model. The upper layer is used for joint decision-making on the task offloading ratio and computing resource allocation, while the lower layer is used for UAV flight trajectory control and formation maintenance. In the high-level hierarchy, each follower drone decides its task offloading ratio based on its own task characteristics, communication conditions, and system load information. The leader drone, sensing the offloading ratios of each follower drone, uniformly allocates computing resources to minimize system task completion latency. The leader drone's state space includes the offloading decisions of each follower drone and global system load information; its action space includes the computing resource allocation strategy for each follower drone; the leader drone's reward function is constructed from system task completion latency, system-level latency improvement rate, and offloading computing power mismatch penalty. The follower drones' state spaces include their own task characteristics, communication conditions, and system load information; their action spaces are task offloading ratios; and their reward functions are constructed from individual task completion latency, individual-level latency improvement rate, system-level latency improvement rate, and bottleneck penalties. In high-altitude environments, the reward function for following the drone is calculated using the following formula: ， ， ， , , , , They are time slots k Time i The reward function value of the drone following the system, the individual task completion latency, the individual-level latency improvement rate, the system-level latency improvement rate, and the bottleneck penalty. For time slots k Time i The mission completion delay of the drone. For time slots k Time i The time required for all local processing by the drone. As the first parameter, For time slots k System latency Tolerance coefficient, , , , These are, respectively, the individual task completion latency weight, the individual-level latency improvement rate weight, the individual-level latency improvement rate weight, and the bottleneck penalty weight. For indicator functions; In the lower layer, each UAV independently outputs continuous flight actions based on local observation information, simultaneously satisfying UAV dynamics constraints, formation constraints, and obstacle avoidance safety constraints during flight. The state space of the leader UAV includes its motion state, relative position to the target, and local obstacle information. The action space of the leader UAV includes velocity increment and heading increment. The reward function of the leader UAV is constructed from distance advancement reward, target arrival reward, time pressure penalty, and first obstacle avoidance and boundary constraint penalty. The state space of the follower UAV includes its motion state, relative position information with the leader UAV, relative position information with neighboring follower UAVs, and local obstacle information. The reward function of the follower UAV is constructed from formation distance reward, velocity and heading coordination reward, neighbor distance and collision avoidance reward, and second obstacle avoidance and boundary constraint penalty. The formula for calculating time pressure penalty is as follows: ， The time pressure penalty value, The maximum number of permissible flight slots, This indicates the threshold time at which time pressure begins to take effect. Time pressure penalty weight; By employing a decoupled two-layer multi-agent reinforcement learning method, the lower and higher layers are trained separately to generate UAV flight trajectories, task offloading decisions, and computational resource allocation strategies, thereby enabling UAV swarm path planning and task offloading.

2. The method for path planning and task offloading of unmanned aerial vehicle swarms oriented towards edge computing as described in claim 1, characterized in that, During the high-level decision-making phase, the follower drones independently decide on the task offloading ratio based on the high-level status, offloading some tasks to the leader drone. Subsequently, the leader drone, based on the offloading decisions of all follower drones, uniformly allocates computing resources.

3. The method for path planning and task offloading of UAV swarms oriented towards edge computing according to claim 2, characterized in that, After completing high-level decision-making, each UAV executes flight maneuvers based on low-level strategies. The lead UAV updates its flight status according to the mission objectives, and the follower UAV adjusts its speed and heading through local observation to maintain the formation structure.

4. The method for path planning and task offloading of UAV swarms oriented towards edge computing according to claim 1, characterized in that, A phased training approach is adopted, first training the lower levels and then training the higher levels.

5. The method for path planning and task offloading of UAV swarms oriented towards edge computing according to claim 4, characterized in that, Low-level training consists of two phases: The first phase only trains the flight control strategy of the leader drone and updates the network parameters corresponding to the leader drone. In the second phase, while freezing the low-level network parameters of the leader drone, the formation control strategy for the follower drones is trained, and only the network parameters corresponding to the follower drones are updated.

6. A drone swarm path planning and task unloading device for edge computing, used to implement the drone swarm path planning and task unloading method for edge computing as described in claim 1, characterized in that, include: The system modeling module is used to construct the UAV swarm-assisted edge computing system into a two-layer heterogeneous multi-agent decision-making model, in which the high layer is used for task offloading and computing resource allocation decisions, and the low layer is used for UAV flight trajectory control and formation keeping. The high-level decision-making module is used to determine the task offloading ratio of each follower drone based on the task characteristics, communication conditions and system load information of the follower drones; and to uniformly allocate the computing resources of the leader drone based on the task offloading ratio to minimize the system task completion delay. The low-level flight control module is used to output continuous flight maneuvers based on local observation information of the UAV; during the execution of flight maneuvers, it simultaneously satisfies the UAV's dynamic constraints, formation constraints, and obstacle avoidance safety constraints. The two-layer reinforcement learning processing module is used to train the lower-level flight control module and the higher-level decision-making module respectively using a decoupled two-layer multi-agent reinforcement learning algorithm, so as to output the UAV flight trajectory, mission offloading decision and computing resource allocation strategy.

7. A path planning and task unloading device for unmanned aerial vehicle (UAV) swarms oriented towards edge computing, characterized in that, It includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the UAV swarm path planning and task offloading method for edge computing as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Used to store computer programs, wherein the computer programs, when executed by a processor, implement the UAV swarm path planning and task offloading method for edge computing as described in any one of claims 1-5.

Images