A method and related device for cooperative collision avoidance of a UAV cluster

Abstract

The application provides a UAV cluster cooperative collision avoidance method and related device, and relates to the technical field of UAV control. The method realizes predictive conflict detection by calculating the nearest approach time and the nearest approach distance, and combines constraint reinforcement learning to optimize the collision avoidance strategy, effectively identifies potential conflict risks in advance, and ensures that the decentralized execution meets the safety constraints, has predictive conflict awareness capability, an explicit safety constraint mechanism and a decentralized real-time execution characteristic.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) control technology, and in particular to a method and device for collaborative collision avoidance of UAV swarms. Background Technology

[0002] Urban low-altitude airspace is characterized by limited flight space, densely distributed obstacles, frequent flight path intersections, and complex airspace operation rules. In this environment, when multiple drones perform logistics delivery, inspection, or emergency rescue missions, they must simultaneously achieve target point arrival and close-range conflict avoidance under limited communication range and local observation conditions. Due to the dynamic relative motion between drones, two drones that are currently far apart may rapidly approach each other within a short period. Relying solely on the current distance for collision avoidance judgment will result in severely delayed conflict warnings, failing to meet real-time safety requirements. Existing collision avoidance technologies mainly fall into three categories: geometric rule-based methods, such as artificial potential field methods or velocity obstacle methods, while possessing advantages in computational efficiency and ease of implementation, are easily constrained by local rules and limited by instantaneous state information in complex 3D scenes and high-density multi-machine interactions, making it difficult to effectively handle future conflict trends; traditional path planning methods, such as online replanning algorithms like A* or RRT, can utilize environmental structures for path search, but place excessive demands on dynamic multi-machine interactive response, real-time replanning capabilities, and onboard computing resources, making them difficult to adapt to the rapidly changing airspace environment of urban low-altitude areas; and ordinary multi-agent reinforcement learning methods, such as MAPPO or MADDPG, can learn decision-making strategies through environmental interaction, but their safety mechanisms typically rely solely on terminal collision penalties or current distance penalties, failing to identify potential conflict risks that are currently not in contact but are about to approach in the future, and lacking explicit expressions of safety constraints and constraints on the training process. It is worth noting that while encounter geometry indicators such as TCPA / DCPA have been applied in maritime and air traffic conflict detection, they have not yet been deeply integrated with the safety constraint training mechanism of multi-UAV constraint reinforcement learning. This results in significant shortcomings of existing technologies in predictive conflict perception, explicit modeling of safety constraints, and distributed real-time execution. Therefore, there is an urgent need to develop a cooperative collision avoidance mechanism that can integrate future conflict trend prediction, explicit training of safety constraints, and distributed execution deployment. This mechanism would enable UAVs to perceive conflict risks in advance in complex and dynamic urban low-altitude environments and output cooperative collision avoidance actions that meet the requirements of real-time control cycles.

[0003] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0004] This application provides a method and related apparatus for collaborative collision avoidance of unmanned aerial vehicle (UAV) swarms, which has predictive conflict perception capabilities, explicit safety constraint mechanisms, and distributed real-time execution characteristics.

[0005] Firstly, the UAV swarm cooperative collision avoidance method provided in this application adopts the following technical solution: Step 1: Model the multi-UAV cooperative collision avoidance problem as a distributed partially observable Markov decision process, defining the system state space, the local observation space of each UAV, and the joint action space; Step 2: For any two drones, calculate the closest approach time and closest approach distance based on their relative position and relative speed. When the closest approach time is not negative and is less than a preset warning time window threshold, and the closest approach distance is less than a preset safe distance threshold, it is determined that there is a potential conflict between the two drones, triggering a predictive conflict penalty term. Aggregate the predictive conflict penalty terms of each drone to its neighboring drones within the communication range to obtain the conflict cost signal of each drone. The conflict cost signal is independent of the cooperative reward function and serves as the cost channel input for constrained reinforcement learning. Step 3: Train the action-evaluation policy network with shared parameters using a centralized training and decentralized execution method. The goal is to maximize the expected cumulative reward and ensure that the expected cumulative conflict cost does not exceed a preset constraint threshold. Update the policy network parameters and use a proportional-integral-derivative control law to adaptively update the constraint multipliers. Step 4: Lightweight the trained motion network and deploy the lightweight motion network to the onboard computing unit of each UAV; during the execution phase, each UAV independently runs the motion network based only on its local observations and outputs collision avoidance control commands in real time.

[0006] Optionally, in step one, the system state space is defined based on the UAV mass dynamics model, and the state of a single UAV includes three-dimensional position, three-dimensional velocity, heading angle, and remaining battery percentage; the local observation space is defined based on the onboard positioning sensor and the inter-UAV communication link; in the joint action space, the action of a single UAV is three-dimensional continuous acceleration; the local observation vector of each UAV consists of its own state, its position vector relative to the target point, and the relative motion state of neighboring UAVs within the communication range; when the number of neighboring UAVs within the communication range exceeds a preset upper limit for neighboring UAV input, the neighboring UAVs are sorted in descending order according to their threat level, and the top-ranked neighboring UAVs are selected; when the number of neighboring UAVs within the communication range is less than the upper limit for neighboring UAV input, the corresponding observation components are supplemented with zero vectors; thus, a local observation vector with a fixed dimension is constructed, making the input dimension of the policy network independent of the cluster size; wherein, the threat level is determined based on the relative distance, relative approach speed, closest approach time, and closest approach distance between the neighboring UAV and the host, the neighboring UAV with the smaller the closest approach time, the smaller the closest approach distance, the closer the relative distance, and the greater the relative approach speed is assigned a higher threat level and is preferentially included in the host's local observation vector.

[0007] Optionally, in step two, when the calculated result of the closest approach time is negative, i.e., the two drones are moving away from each other, the closest approach time is treated as zero, and the closest approach distance degenerates into the current relative distance between the two drones; the predictive conflict penalty term takes a value positively correlated with the preset penalty weight when the potential conflict triggering condition is met, and takes zero when the potential conflict triggering condition is not met; the aggregation is the summation of the predictive conflict penalty terms of each neighboring drone within the communication range, or taking the maximum value among them; the predictive conflict penalty term is also implemented using a continuous or piecewise continuous penalty function that increases as the closest approach time and the closest approach distance decrease, and the penalty function includes an exponentially decaying penalty function.

[0008] Optionally, in step two, the cooperative reward function consists of a task completion reward and a flight efficiency penalty term; the flight efficiency penalty term includes energy consumption cost, maneuver smoothness cost, and time consumption cost, and each cost component is a non-negative value. The task completion reward and the flight efficiency penalty term are multiplied by their respective weight coefficients and then combined to form the cooperative reward function.

[0009] Optionally, in step three, during the training phase, the evaluation network in the action-evaluation policy network estimates the joint state value using information containing the global states of multiple UAVs; the cumulative reward expectation in the constrained optimization objective is a discounted cumulative reward expectation, and the cumulative conflict cost expectation is a discounted cumulative conflict cost expectation; the constrained optimization objective is transformed into an unconstrained optimization objective containing constrained multipliers through Lagrange relaxation.

[0010] Optionally, in step three, the conflict rate constraint violation is the difference between the measured cumulative conflict cost expectation and the preset constraint threshold; in the proportional-integral-derivative control law, the proportional term is proportional to the current conflict rate constraint violation, the integral term is proportional to the historical cumulative amount of the conflict rate constraint violation, and the derivative term is proportional to the difference between two adjacent conflict rate constraint violations; the three terms are weighted and summed according to their respective proportional coefficient, integral coefficient, and derivative coefficient, and then multiplied by the preset multiplier update step size, and then accumulated and updated to the constraint multiplier, and the updated constraint multiplier is constrained to a non-negative value.

[0011] Optionally, in step three, the training process adopts a course learning strategy from simple to difficult: the training scenario transitions sequentially from a low-interaction-density scenario to a high-density multi-machine conflict scenario; after the current training stage meets the preset stage switching conditions, it enters the next training stage. The stage switching conditions are reaching the preset training round budget, or the success rate of the current stage reaches the preset stage success rate threshold.

[0012] Optionally, in step four, the lightweighting process includes performing 8-bit integer symmetric parameter quantization on the weight matrix of the action network: the quantization scaling factor is determined by the ratio of the maximum absolute value of the elements in the weight matrix to the range of the 8-bit integer representation, and the weight matrix is ​​converted from floating-point representation to 8-bit integer representation for storage, reducing the storage footprint of the policy network to about one-quarter of that of floating-point storage; and the input dimension of the action network is fixed by a preset upper limit for neighbor input, so that the computational cost of single-step forward inference is independent of the cluster size; the local observation construction method in the execution phase is consistent with the local observation construction method in the training phase; the collision avoidance control command is a three-dimensional continuous acceleration control command, which is converted into an execution command through the flight control interface; in each control cycle of the execution phase, each UAV reads its own status, obtains neighbor information within the communication range, constructs a local observation vector with fixed dimensions, and outputs the three-dimensional continuous acceleration control command through forward inference of the action network.

[0013] Optionally, the warning time window threshold is on the order of ten seconds, the safety distance threshold is on the order of fifteen meters, the preset constraint threshold is determined based on a preset upper limit of the conflict rate, and the upper limit of the conflict rate is on the order of five percent; the discount factor between the expected cumulative reward and the expected cumulative conflict cost is on the order of 0.99.

[0014] Secondly, this application also provides a drone swarm cooperative collision avoidance device, comprising: The spatial module is used to model the multi-UAV cooperative collision avoidance problem as a distributed partially observable Markov decision process, defining the system state space, the local observation space of each UAV, and the joint action space. The calculation module is used to calculate the closest approach time and closest approach distance between any two drones based on their relative position and relative speed. When the closest approach time is not negative and is less than a preset warning time window threshold, and the closest approach distance is less than a preset safe distance threshold, it is determined that there is a potential conflict between the two drones, triggering a predictive conflict penalty term. The predictive conflict penalty terms of each drone to its neighboring drones within the communication range are aggregated to obtain the conflict cost signal of each drone. The conflict cost signal is independent of the cooperative reward function and serves as the cost channel input for constrained reinforcement learning. The network module is used to train the action-evaluation policy network with shared parameters using a centralized training and decentralized execution approach. The optimization objective is to maximize the expected cumulative reward and ensure that the expected cumulative conflict cost does not exceed a preset constraint threshold. The module updates the policy network parameters and uses a proportional-integral-derivative control law to adaptively update the constraint multipliers. The output module is used to perform lightweight processing on the trained action network and deploy the lightweight action network to the onboard computing unit of each UAV. During the execution phase, each UAV independently runs the action network based only on its local observations and outputs collision avoidance control commands in real time.

[0015] In summary, this application achieves predictive conflict detection by calculating the nearest approach time and nearest approach distance, and optimizes the collision avoidance strategy by combining constraint reinforcement learning. This effectively identifies potential conflict risks in advance, while ensuring that distributed execution meets security constraints. It has predictive conflict perception capabilities, explicit security constraint mechanisms, and distributed real-time execution characteristics. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the first embodiment of the UAV swarm cooperative collision avoidance method of this application; Figure 2 This is a flowchart of TCPA / DCPA calculation and conflict penalty judgment in the first embodiment of the UAV swarm cooperative collision avoidance method of this application; Figure 3 This is a flowchart of the constraint reinforcement learning training process of the first embodiment of the UAV swarm cooperative collision avoidance method of this application; Figure 4 This is an airborne deployment flowchart of the first embodiment of the UAV swarm cooperative collision avoidance method of this application; Figure 5 This is a single-step real-time decision-making flowchart of the first embodiment of the UAV swarm cooperative collision avoidance method of this application; Figure 6 This is a structural block diagram of the first embodiment of the drone swarm cooperative collision avoidance device of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] Traditional UAV collision avoidance methods face challenges in complex and dynamic urban low-altitude environments, including limited flight space, dense obstacles, and frequent path intersections. Geometric rule-based methods are limited by local rules and the current state; traditional path planning methods require high levels of dynamic interaction and computational resources; and ordinary multi-agent reinforcement learning methods struggle to identify potential conflicts that are not currently colliding but are approaching in the future, and lack explicit expressions of safety constraints, resulting in delayed conflict warnings and failing to meet real-time control requirements.

[0019] To address this, this application provides a method for cooperative collision avoidance by a swarm of unmanned aerial vehicles (UAVs), referring to... Figure 1 ,Figure 1 This is a flowchart illustrating the first embodiment of the drone swarm cooperative collision avoidance method of this application.

[0020] In this embodiment, the drone swarm cooperative collision avoidance method includes the following steps: Step 1: Model the multi-UAV cooperative collision avoidance problem as a distributed partially observable Markov decision process, defining the system state space, the local observation space of each UAV, and the joint action space.

[0021] For ease of understanding, the following explains some key terms in this embodiment: Distributed Partially Observable Markov Decision Processes (Dec-POMDPs) are a mathematical framework for modeling multi-agent decision-making problems, where each agent can only obtain local observation information and needs to make decisions based on this information to achieve a global objective. This framework is applicable to the cooperative collision avoidance problem of UAV swarms under limited communication and observation conditions.

[0022] The system state space defines the set of all possible states of a drone swarm at a given moment, including information such as the position and velocity of each drone. The local observation space describes the environmental information that a single drone can acquire, such as its own state and relative information about neighboring drones within its communication range. The joint action space refers to the combination of actions that all drones in the swarm can take at a given moment.

[0023] Time to Closest Approach (TCPA) and Distance to Closest Approach (DCPA) are geometric metrics used to assess the potential risk of future conflict between two drones. TCPA represents the time required for two drones to reach their closest distance to each other under their current motion trends, while DCPA represents the minimum distance between the two drones at that point in time.

[0024] A predictive conflict penalty is a signal triggered before a potential conflict occurs, used to guide drones to avoid conflict during reinforcement learning training. The penalty is triggered based on calculations of the closest approach time and closest approach distance; when these metrics meet preset conflict conditions, it indicates the presence of a potential conflict.

[0025] The conflict cost signal is obtained by aggregating the predictive conflict penalties of each UAV for its neighbors within its communication range. It is used to quantify the conflict risk at the current moment. This signal serves as the cost channel input for constrained reinforcement learning, independent of the cooperative reward function, to ensure the explicit expression of safety constraints.

[0026] Constrained reinforcement learning is a reinforcement learning paradigm that maximizes the expected cumulative reward while ensuring that the expected cumulative conflict cost does not exceed a preset constraint threshold. This method introduces cost channels and constrained optimization objectives, enabling the agent to strictly adhere to safety restrictions when learning the optimal policy.

[0027] Centralized training and distributed execution is a training paradigm for multi-agent reinforcement learning. During the training phase, global information is used to train the policy network to improve learning efficiency and policy performance; during the execution phase, each agent independently runs the trained policy network based only on its local observations, achieving efficient distributed decision-making.

[0028] Parameter-sharing action-evaluation policy networks are a type of deep reinforcement learning network architecture where the action network outputs action instructions based on observations, and the evaluation network assesses the value of the current state. The parameter-sharing mechanism improves training efficiency and generalization ability when dealing with multi-agent problems.

[0029] The proportional-integral-derivative (PID) control law is a classic feedback control algorithm used to adaptively adjust the control input based on system error. In this method, this control law is used to adaptively update the constraint multipliers to dynamically adjust the strength of safety constraints in constraint reinforcement learning.

[0030] Lightweight processing refers to optimizing the trained action network to reduce its model size and computational complexity, enabling it to be efficiently deployed on resource-constrained hardware platforms such as onboard computing units to meet the needs of real-time inference.

[0031] It should be noted that in step one, the system state space is defined based on the UAV mass dynamics model, and the state of a single UAV includes three-dimensional position, three-dimensional velocity, heading angle, and remaining battery percentage; the local observation space is defined based on the onboard positioning sensor and the inter-UAV communication link; in the joint action space, the action of a single UAV is three-dimensional continuous acceleration; the local observation vector of each UAV consists of its own state, its position vector relative to the target point, and the relative motion state of neighboring UAVs within the communication range; when the number of neighboring UAVs within the communication range exceeds the preset upper limit of neighboring UAV input, the system is adjusted according to the threat level. After neighboring machines are sorted in descending order, the top-ranked neighboring machines are selected. When the number of neighboring machines within the communication range is less than the upper limit of the neighboring machine input, the corresponding observation components are supplemented with zero vectors. This constructs a local observation vector with a fixed dimension, making the input dimension of the policy network independent of the cluster size. The threat level is determined based on the relative distance, relative approach speed, closest approach time, and closest approach distance between the neighboring machine and the local machine. The smaller the closest approach time, the smaller the closest approach distance, the closer the relative distance, and the greater the relative approach speed, the higher the threat level assigned to the neighboring machine, and the more likely it is to be included in the local observation vector of the local machine.

[0032] In practical implementation, Dec-POMDP is used to formally model the urban low-altitude multi-UAV cooperative collision avoidance problem. The multi-UAV swarm cooperative collision avoidance problem is modeled as a distributed partially observable Markov decision process. The system state space is defined based on the UAV particle dynamics model, the local observation space of each agent is defined based on local sensors and inter-Agent communication, and the joint action space is defined based on continuous acceleration control. Specifically, each UAV is approximated as a particle dynamics model, and the state vector is defined as a combination of position, velocity, heading angle and remaining battery power, as shown in Equation (1): in, Indicates the first A drone in The three-dimensional position at a given moment. Represents three-dimensional velocity. Indicates the heading angle. This represents the remaining battery percentage. The motion vector is defined as a three-dimensional continuous acceleration component, as shown in equation (2): The state transition is updated according to the UAV particle dynamics equation, as shown in equation (3): The heading angle is derived from the horizontal component of the updated velocity, and the remaining battery power is updated according to the onboard energy consumption model as it decreases during flight. Neither of these components is considered an independently controlled dynamic component.

[0033] will contain The swarm cooperative collision avoidance problem of UAVs is formally defined as a seven-tuple Dec-POMDP, as shown in equation (4): in, Represents a collection of unmanned aerial vehicle (UAV) intelligent agents. Represents the global state space. Indicates the first The operational space of the drone This represents the state transition probability function. Indicates the first The local observation space of the UAV (the observation is generated by the deterministic observation mapping defined by equation (5) below), Represents the reward function, This represents the discount factor.

[0034] Construct the local observation vector for each UAV. The local observation vector consists of the local state, the local position vector relative to the target point, and the relative motion state of neighboring UAVs within the communication range, as shown in equation (5): in, Indicates drone The target point position vector, Indicates drone exist The set of neighboring machines within the communication range at any given time.

[0035] When the number of neighboring devices within the communication range exceeds the limit At that time, neighboring machines are sorted in descending order according to threat level, and the first few are extracted. There are several neighboring machines; when the number of neighboring machines is less than the upper limit, the corresponding observation components are padded with zero vectors. This constructs a fixed-dimensional local observation vector to ensure that the input dimension of the policy network is independent of the cluster size.

[0036] Through the above technical solution, this embodiment provides a concrete and efficient modeling foundation for the collaborative collision avoidance problem of UAV swarms. The system state space is defined based on the UAV mass dynamics model, incorporating key information such as heading angle and remaining battery power ratio. This allows the model to fully reflect the motion characteristics and mission constraints of the UAVs while maintaining computational efficiency. Simultaneously, a local observation space is constructed through onboard positioning sensors and inter-UAV communication links, ensuring that each UAV can acquire necessary local situational information. More importantly, this embodiment cleverly designs the local observation vectors so that their input dimension is independent of the swarm size. Specifically, by comprehensively considering factors such as relative distance, relative approach speed, closest approach time, and closest approach distance, the threat level of neighboring UAVs within the communication range is assessed and ranked, and then truncated or supplemented according to a preset upper limit, thereby constructing a local observation vector with a fixed dimension. This design enables the trained policy network to be generalized to UAV swarms of different sizes, greatly improving the robustness, scalability, and practicality of the collision avoidance strategy. It effectively solves the problem of inconsistent input dimensions of the policy network in variable swarm environments, ensuring the real-time performance and effectiveness of collision avoidance control commands.

[0037] Step 2: For any two drones, calculate the closest approach time and closest approach distance based on their relative position and relative speed. When the closest approach time is not negative and is less than a preset warning time window threshold, and the closest approach distance is less than a preset safe distance threshold, it is determined that there is a potential conflict between the two drones, triggering a predictive conflict penalty term. Aggregate the predictive conflict penalty terms of each drone to its neighboring drones within the communication range to obtain the conflict cost signal of each drone. The conflict cost signal is independent of the cooperative reward function and serves as the cost channel input for constrained reinforcement learning.

[0038] It should be noted that in step two, when the calculated result of the closest approach time is negative, that is, when the two drones are moving away from each other, the closest approach time is treated as zero, and the closest approach distance degenerates into the current relative distance between the two drones; the predictive conflict penalty term takes a value positively correlated with the preset penalty weight when the potential conflict triggering condition is met, and takes zero when the potential conflict triggering condition is not met; the aggregation is the summation of the predictive conflict penalty terms of each neighboring drone within the communication range of the local machine, or taking the maximum value among them; the predictive conflict penalty term is also implemented using a continuous or piecewise continuous penalty function that increases as the closest approach time and the closest approach distance decrease, and the penalty function includes an exponential decay form penalty function.

[0039] By employing the aforementioned technical solution, when drones are moving away from each other, the nearest approach time is treated as zero and the nearest approach distance is degraded to the current relative distance. This allows for an accurate assessment of the actual approach risk in the current state, avoiding ineffective responses to resolved or non-existent conflicts. Simultaneously, by introducing a continuous or piecewise continuous penalty function that increases with decreasing nearest approach time and nearest approach distance, particularly an exponentially decaying penalty function, the predictive conflict penalty term provides a refined and gradient-based conflict risk signal. This smooth and responsive penalty mechanism provides a more instructive cost channel input for constrained reinforcement learning, enabling the policy network to learn more accurate and robust collision avoidance strategies. This significantly improves the collaborative collision avoidance capability and safety of drone swarms in complex dynamic environments while ensuring task completion efficiency. Furthermore, the flexible aggregation method allows the system to better adapt to swarm scenarios with varying densities.

[0040] It is understood that in step two, the cooperative reward function consists of a task completion reward and a flight efficiency penalty term; the flight efficiency penalty term includes energy consumption cost, maneuver smoothness cost and time consumption cost, and each cost component is a non-negative value. The task completion reward and the flight efficiency penalty term are multiplied by their respective weight coefficients and then combined to form the cooperative reward function.

[0041] Through the above technical solution, this embodiment can effectively guide the policy network to pursue task completion while taking into account the economy, stability, and timeliness of the actual flight of the UAV. Specifically, the introduction of energy consumption cost prompts the UAV to choose more energy-efficient flight paths and speeds during collision avoidance, thereby extending the UAV's endurance and improving the continuous operation capability of the task; the introduction of maneuver smoothness cost avoids violent and unnecessary maneuvers, improving flight comfort and safety, while reducing mechanical wear and extending the service life of the UAV; time consumption cost encourages the UAV to complete the task in a more efficient manner, shortening the overall operation time and improving the response speed and completion efficiency of the task. All cost components are non-negative and are flexibly combined through weight coefficients, so that the trained strategy can not only successfully avoid collisions, but also achieve a better balance among multiple targets, significantly improving the overall operational performance and practicality of the UAV swarm.

[0042] The warning time window threshold is on the order of ten seconds, the safety distance threshold is on the order of fifteen meters, the preset constraint threshold is determined based on the preset upper limit of the conflict rate, and the upper limit of the conflict rate is on the order of five percent; the discount factor between the expected cumulative reward and the expected cumulative conflict cost is on the order of 0.99.

[0043] In practical implementation, a predictive conflict penalty and cooperative reward function based on the TCPA / DCPA geometric conflict prediction model is designed. The closest approach time (TCPA) and closest approach distance (DCPA) between any two UAVs are calculated to construct a predictive conflict penalty term, enabling early detection and penalty of potential conflicts. The cooperative reward function includes a task completion reward and a flight efficiency penalty term. The predictive conflict penalty term, after aggregation by neighboring UAVs, serves as an independent conflict cost signal, which is input into the cost channel of the constraint reinforcement learning, specifically including: Set up drones With drones The relative position vector is The relative velocity vector is As shown in equations (6) and (7) respectively: Calculate the closest approach time As shown in equation (8): Calculate the nearest proximity distance As shown in equation (9): in, To prevent positive numbers with a denominator of zero, when the result of equation (8) is negative, i.e., when the two machines are moving away from each other, [the following will be done]. Treat it as 0, then equation (9) gives It degenerates into the current relative distance between the two machines.

[0044] when Greater than or equal to zero and less than the warning time window threshold ,and Less than the safety distance threshold At that time, the drone was determined With drones There is a potential conflict, triggering a predictive conflict penalty term, and the judgment condition is shown in equation (10). The TCPA / DCPA conflict prediction process is as follows: Figure 2 As shown.

[0045] The predictive conflict penalty term is expressed as equation (11): in, This indicates the penalty weight. If the potential conflict triggering condition is not met, then let... .

[0046] The collaborative reward function consists of a task completion reward and a flight efficiency penalty term, as shown in equation (12): in, This indicates the reward for completing the task. This represents a flight efficiency penalty, which includes non-negative cost components such as energy consumption, maneuverability smoothness, and time consumption. , These represent the weighting coefficients.

[0047] The predictive conflict penalty term is not included in the collaborative reward function, but is aggregated by neighboring machines to form the conflict cost signal of each UAV, which is used as the cost channel input of the conflict rate constraint in the constrained reinforcement learning and is used to form the expected cumulative conflict cost in equation (13).

[0048] Step 3: Train the action-evaluation policy network with shared parameters using a centralized training and decentralized execution approach. The goal is to maximize the expected cumulative reward and ensure that the expected cumulative conflict cost does not exceed a preset constraint threshold. Update the policy network parameters and use a proportional-integral-derivative control law to adaptively update the constraint multipliers.

[0049] It should be noted that in step three, during the training phase, the evaluation network in the action-evaluation policy network estimates the joint state value using information containing the global states of multiple UAVs; the cumulative reward expectation in the constrained optimization objective is a discounted cumulative reward expectation, and the cumulative conflict cost expectation is a discounted cumulative conflict cost expectation; the constrained optimization objective is transformed into an unconstrained optimization objective containing constrained multipliers through Lagrange relaxation.

[0050] Through the above technical solutions, during the training phase, the evaluation network utilizes information encompassing the global states of multiple drones to estimate the joint state value. This enables a more comprehensive and accurate assessment of the current situation and potential risks of the entire swarm, providing high-quality value guidance for the action network's policy updates and avoiding suboptimal decisions due to insufficient local observations. Simultaneously, setting the expected cumulative reward and expected cumulative conflict cost in a discounted form allows the model to better balance short-term gains with long-term goals during learning, enhancing the stability and foresight of the policy. Furthermore, by using Lagrange relaxation to transform the constrained optimization objective into an unconstrained optimization objective with constrained multipliers, the optimization process is greatly simplified, enabling the efficient solution of complex constrained reinforcement learning problems. This ensures that while maximizing task completion rewards, the conflict cost of the drone swarm is strictly controlled, effectively improving the convergence speed of training and the safety of the policy.

[0051] Understandably, in step three, the conflict rate constraint violation is the difference between the measured cumulative conflict cost expectation and the preset constraint threshold; in the proportional-integral-derivative control law, the proportional term is proportional to the current conflict rate constraint violation, the integral term is proportional to the historical cumulative amount of the conflict rate constraint violation, and the derivative term is proportional to the difference between two adjacent conflict rate constraint violations; the three terms are weighted and summed according to their respective proportional coefficient, integral coefficient, and derivative coefficient, and then multiplied by the preset multiplier update step size, and then accumulated and updated to the constraint multiplier, and the updated constraint multiplier is constrained to a non-negative value.

[0052] The above technical solution defines the conflict rate constraint violation as the difference between the measured cumulative conflict cost expectation and the preset constraint threshold, and uses a proportional-integral-derivative control law to adaptively update the constraint multiplier. Specifically, the proportional term can quickly respond to the current constraint violation level and adjust the constraint multiplier in a timely manner; the integral term effectively eliminates long-term steady-state errors by accumulating historical violations, enhancing the robustness of the system; and the derivative term can predict the changing trend of the violation amount and adjust it in advance, thereby suppressing oscillations and accelerating convergence. The three terms are weighted, summed, multiplied by the update step size, and added to the constraint multiplier, ensuring its non-negativity, making the adjustment process of the constraint multiplier smoother, more stable, and more efficient. This effectively solves the problem of how to accurately and robustly adjust the constraint multiplier to ensure that the cumulative conflict cost expectation always meets the preset constraint threshold in UAV swarm cooperative collision avoidance training, avoiding oscillations or slow convergence during the training process, so that the trained collision avoidance strategy can effectively avoid conflicts and maintain high task efficiency.

[0053] It should be noted that in step three, the training process adopts a course learning strategy from simple to difficult: the training scenarios transition sequentially from low-interaction density scenarios to high-density multi-machine conflict scenarios; after the current training stage meets the preset stage switching conditions, it enters the next training stage. The stage switching conditions are reaching the preset training round budget, or the success rate of the current stage reaches the preset stage success rate threshold.

[0054] In practical implementation, a centralized training-decentralized execution paradigm and a parameter-sharing Actor-Critic architecture are adopted to train the multi-agent policy network. During the training phase, global state information is used to optimize the centralized Critic network, and the Lagrange constraint multipliers are adaptively adjusted through a constrained proximal policy optimization algorithm and a PID control law. During the execution phase, each UAV independently runs the Actor network based solely on its local observations, outputting decision actions, specifically including: A parameter-sharing Actor-Critic network architecture is constructed, in which all UAVs share the same set of Actor policy network parameters; during the training phase, a centralized Critic network is set up, which uses information containing the global state of multiple UAVs to estimate the joint state value; during the execution phase, each UAV independently runs the Actor network to output continuous actions based only on its local observations.

[0055] The constrained proximal policy optimization algorithm is adopted to maximize the expected cumulative reward while satisfying the upper limit constraint on the conflict rate. The constraint optimization objective is shown in equation (13): in, Indicates the policy network parameters, Indicates the expected cumulative reward. This indicates the expectation of accumulating conflict costs. This represents the safety constraint threshold. Equation (13) is transformed into Lagrangian form, as shown in Equation (14): in, For Lagrange-bound multipliers.

[0056] The Lagrange constraint multipliers are adaptively updated using a PID control law. The constraint violation is defined as shown in equation (15): The three PID terms are shown in equation (16): The Lagrange constraint multiplier update rule is shown in equation (17): in, 、 、 These represent the proportional, integral, and differential coefficients, respectively. In practical implementation, further adjustments can be made to... P, I, D The sum of the three terms is multiplied by the multiplier to update the step size, and then the sum is accumulated and updated.

[0057] The training process employs a learning strategy that progresses from simple to complex, transitioning sequentially from low-interaction-density scenarios to high-density multi-machine conflict scenarios. Each stage proceeds to the next stage after meeting preset stage switching conditions, which are either a preset training round budget or a preset stage success rate threshold. The training process is as follows: Figure 3 As shown.

[0058] Step 4: Lightweight the trained motion network and deploy the lightweight motion network to the onboard computing unit of each UAV; during the execution phase, each UAV independently runs the motion network based only on its local observations and outputs collision avoidance control commands in real time.

[0059] It is understandable that in step four, the lightweight processing includes performing 8-bit integer symmetric parameter quantization on the weight matrix of the action network: the quantization scaling factor is determined by the ratio of the maximum absolute value of the elements in the weight matrix to the range of the 8-bit integer representation, and the weight matrix is ​​converted from floating-point representation to 8-bit integer representation for storage, reducing the storage footprint of the policy network to about one-quarter of that of floating-point storage; and the input dimension of the action network is fixed by a preset upper limit for neighbor input, so that the computational cost of single-step forward inference is independent of the cluster size; the local observation construction method in the execution phase is consistent with the local observation construction method in the training phase; the collision avoidance control command is a three-dimensional continuous acceleration control command, which is converted into an execution command through the flight control interface; in each control cycle of the execution phase, each UAV reads its own status, obtains neighbor information within the communication range, constructs a local observation vector with fixed dimensions, and outputs the three-dimensional continuous acceleration control command through the forward inference of the action network.

[0060] By quantizing the trained action network with 8-bit integer symmetric parameters and fixing its input dimensions, the storage footprint of the policy model and the computational cost of single-step forward inference are significantly reduced. This enables the efficient deployment of complex collision avoidance policy networks on resource-constrained airborne computing units, effectively solving the deployment challenges in real-world applications with high real-time requirements and limited computing resources. Simultaneously, the consistent local observation construction method between the execution and training phases ensures the model's stability and performance in actual operation. Within each control cycle, each UAV can quickly read its own status, acquire neighboring UAV information, construct fixed-dimensional observation vectors, and output three-dimensional continuous acceleration control commands in real time through the lightweight action network. This achieves low-latency, high-efficiency autonomous collision avoidance decision-making, greatly improving the collaborative collision avoidance capabilities and task execution efficiency of UAV swarms in complex environments.

[0061] In practice, the trained Actor network undergoes lightweighting. This lightweighting process includes performing INT8 symmetric parameter quantization on the Actor network weight matrix. Let the weight matrix be... Quantization scaling factor is ,but Calculate according to formula (18): The INT8 quantization weights and the approximate recovery weights during inference are shown in Equation (19): Set an upper limit on the number of inputs from neighboring machines. When the number of neighboring devices within the communication range exceeds At that time, the first part is extracted after sorting by threat level. When the number of neighboring machines is insufficient to reach the upper limit, zero vectors are used to fill the gap, so that the input dimension of the Actor network is fixed as Equation (20). This observation construction mechanism remains consistent during the training and execution phases, and the input scale of onboard inference is limited during deployment.

[0062] in, This represents the dimension of the local observation component (including the local state and the relative target point position vector). This represents the feature dimension of a single neighboring machine.

[0063] The lightweighted Actor network parameters are deployed to the UAV's onboard computing unit. During execution, each UAV reads its own state and neighboring UAV information, constructs a local observation vector, and outputs continuous acceleration control commands through forward inference via the Actor network. The onboard deployment process is as follows: Figure 4 As shown, the single-step real-time decision-making process is as follows: Figure 5 As shown.

[0064] This embodiment achieves predictive conflict detection by calculating the nearest approach time and nearest approach distance, and combines it with constraint reinforcement learning to optimize the collision avoidance strategy, effectively identifying potential conflict risks in advance, while ensuring that distributed execution meets security constraints. It has predictive conflict perception capabilities, explicit security constraint mechanisms, and distributed real-time execution characteristics.

[0065] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the drone swarm cooperative collision avoidance device of this application.

[0066] like Figure 6 As shown, the drone swarm cooperative collision avoidance device proposed in this application includes: Space module 10 is used to model the multi-UAV cooperative collision avoidance problem as a distributed partially observable Markov decision process, defining the system state space, the local observation space of each UAV, and the joint action space. The calculation module 20 is used to calculate the closest approach time and closest approach distance between any two UAVs based on their relative position and relative speed. When the closest approach time is not negative and is less than a preset warning time window threshold, and the closest approach distance is less than a preset safety distance threshold, it is determined that there is a potential conflict between the two UAVs, triggering a predictive conflict penalty term. The predictive conflict penalty terms of each UAV to its neighboring UAV within the communication range are aggregated to obtain the conflict cost signal of each UAV. The conflict cost signal is independent of the cooperative reward function and serves as the cost channel input for constrained reinforcement learning. Network module 30 is used to train the action-evaluation policy network with shared parameters using a centralized training and decentralized execution method. The objective of the constraint optimization is to maximize the expected cumulative reward and ensure that the expected cumulative conflict cost does not exceed a preset constraint threshold. The network parameters are updated, and the constraint multipliers are adaptively updated using a proportional-integral-derivative control law. Output module 40 is used to perform lightweight processing on the trained action network and deploy the lightweight action network to the onboard computing unit of each UAV; during the execution phase, each UAV independently runs the action network based only on its local observations and outputs collision avoidance control commands in real time.

[0067] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solution of this application. In specific applications, those skilled in the art can make settings as needed, and this application does not impose any restrictions on this.

[0068] This embodiment achieves predictive conflict detection by calculating the nearest approach time and nearest approach distance, and combines it with constraint reinforcement learning to optimize the collision avoidance strategy, effectively identifying potential conflict risks in advance, while ensuring that distributed execution meets security constraints. It has predictive conflict perception capabilities, explicit security constraint mechanisms, and distributed real-time execution characteristics.

[0069] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this application. In practical applications, those skilled in the art can select some or all of it to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0070] In addition, for technical details not described in detail in this embodiment, please refer to the method of drone swarm cooperative collision avoidance provided in any embodiment of this application, which will not be repeated here.

[0071] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0072] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0073] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application. The above are only preferred embodiments of this application and do not limit the patent scope of this application. All equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for cooperative collision avoidance in a swarm of unmanned aerial vehicles (UAVs), characterized in that, Performed by a computer, including the following steps: Step 1: Model the multi-UAV cooperative collision avoidance problem as a distributed partially observable Markov decision process, defining the system state space, the local observation space of each UAV, and the joint action space; Step 2: For any two drones, calculate the closest approach time and closest approach distance based on their relative position and relative speed. When the closest approach time is not negative and is less than a preset warning time window threshold, and the closest approach distance is less than a preset safe distance threshold, it is determined that there is a potential conflict between the two drones, triggering a predictive conflict penalty term. Aggregate the predictive conflict penalty terms of each drone to its neighboring drones within the communication range to obtain the conflict cost signal of each drone. The conflict cost signal is independent of the cooperative reward function and serves as the cost channel input for constrained reinforcement learning. Step 3: Train the action-evaluation policy network with shared parameters using a centralized training and decentralized execution method. The goal is to maximize the expected cumulative reward and ensure that the expected cumulative conflict cost does not exceed a preset constraint threshold. Update the policy network parameters and use a proportional-integral-derivative control law to adaptively update the constraint multipliers. Step 4: Lightweight the trained motion network and deploy the lightweight motion network to the onboard computing unit of each UAV; during the execution phase, each UAV independently runs the motion network based only on its local observations and outputs collision avoidance control commands in real time.

2. The UAV swarm cooperative collision avoidance method according to claim 1, characterized in that, In step one, the system state space is defined based on the UAV mass dynamics model. The state of a single UAV includes three-dimensional position, three-dimensional velocity, heading angle, and remaining battery percentage. The local observation space is defined based on the onboard positioning sensor and the inter-UAV communication link. In the joint action space, the action of a single UAV is three-dimensional continuous acceleration. The local observation vector of each UAV consists of its own state, its position vector relative to the target point, and the relative motion state of neighboring UAVs within the communication range. When the number of neighboring UAVs within the communication range exceeds a preset upper limit for neighboring UAV input, the neighboring UAVs are sorted in descending order according to their threat level, and the top-ranked neighboring UAVs are selected. When the number of neighboring UAVs within the communication range is less than the upper limit for neighboring UAV input, the corresponding observation components are supplemented with zero vectors. This constructs a local observation vector with a fixed dimension, making the input dimension of the policy network independent of the cluster size. The threat level is determined based on the relative distance, relative approach speed, closest approach time, and closest approach distance between the neighboring UAV and the host. The closer the closest approach time, the closer the closest approach distance, and the greater the relative approach speed, the higher the threat level assigned to the neighboring UAV, and the more preferentially it is included in the host's local observation vector.

3. The UAV swarm cooperative collision avoidance method according to claim 1, characterized in that, In step two, when the calculated closest approach time is negative, meaning the two drones are moving away from each other, the closest approach time is treated as zero, and the closest approach distance degenerates into the current relative distance between the two drones. The predictive conflict penalty term takes a value positively correlated with the preset penalty weight when the potential conflict triggering condition is met, and takes zero when the potential conflict triggering condition is not met. The aggregation is the summation of the predictive conflict penalty terms of each neighboring drone within the communication range, or taking the maximum value among them. The predictive conflict penalty term is also implemented using a continuous or piecewise continuous penalty function that increases as the closest approach time and the closest approach distance decrease, and the penalty function includes an exponentially decaying penalty function.

4. The UAV swarm cooperative collision avoidance method according to claim 1, characterized in that, In step two, the cooperative reward function consists of a task completion reward and a flight efficiency penalty term. The flight efficiency penalty term includes energy consumption cost, maneuver smoothness cost, and time consumption cost, and each cost component is a non-negative value. The task completion reward and the flight efficiency penalty term are multiplied by their respective weight coefficients and then combined to form the cooperative reward function.

5. The UAV swarm cooperative collision avoidance method according to claim 1, characterized in that, In step three, during the training phase, the evaluation network in the action-evaluation policy network estimates the joint state value using information containing the global states of multiple UAVs; the cumulative reward expectation in the constrained optimization objective is a discounted cumulative reward expectation, and the cumulative conflict cost expectation is a discounted cumulative conflict cost expectation; the constrained optimization objective is transformed into an unconstrained optimization objective containing constrained multipliers through Lagrange relaxation.

6. The UAV swarm cooperative collision avoidance method according to claim 1, characterized in that, In step three, the conflict rate constraint violation is the difference between the measured cumulative conflict cost expectation and the preset constraint threshold. In the proportional-integral-derivative control law, the proportional term is proportional to the current conflict rate constraint violation, the integral term is proportional to the historical cumulative amount of the conflict rate constraint violation, and the derivative term is proportional to the difference between two adjacent conflict rate constraint violations. The three terms are weighted and summed according to their respective proportional coefficient, integral coefficient, and derivative coefficient, and then multiplied by the preset multiplier update step size. The sum is then added to the constraint multiplier, and the updated constraint multiplier is constrained to a non-negative value.

7. The UAV swarm cooperative collision avoidance method according to claim 1, characterized in that, In step three, the training process adopts a course learning strategy from simple to difficult: the training scenarios transition from low-interaction density scenarios to high-density multi-machine conflict scenarios in sequence; after the current training stage meets the preset stage switching conditions, it enters the next training stage. The stage switching conditions are reaching the preset training round budget, or the success rate of the current stage reaches the preset stage success rate threshold.

8. The UAV swarm cooperative collision avoidance method according to claim 1, characterized in that, In step four, the lightweighting process includes performing 8-bit integer symmetric parameter quantization on the weight matrix of the action network: the quantization scaling factor is determined by the ratio of the maximum absolute value of the elements in the weight matrix to the range of the 8-bit integer representation, and the weight matrix is ​​converted from floating-point representation to 8-bit integer representation for storage, reducing the storage footprint of the policy network to about one-quarter of that of floating-point storage; and the input dimension of the action network is fixed by a preset upper limit for neighbor input, so that the computational cost of single-step forward inference is independent of the cluster size; the local observation construction method in the execution phase is consistent with that in the training phase; the collision avoidance control command is a three-dimensional continuous acceleration control command, which is converted into an execution command through the flight control interface; in each control cycle of the execution phase, each UAV reads its own status, obtains neighbor information within the communication range, constructs a local observation vector with fixed dimensions, and outputs the three-dimensional continuous acceleration control command through forward inference of the action network.

9. The UAV swarm cooperative collision avoidance method according to claim 1, characterized in that, The warning time window threshold is on the order of ten seconds, the safety distance threshold is on the order of fifteen meters, the preset constraint threshold is determined based on the preset upper limit of the conflict rate, and the upper limit of the conflict rate is on the order of five percent; the discount factor between the expected cumulative reward and the expected cumulative conflict cost is on the order of 0.

99.

10. A collaborative collision avoidance device for unmanned aerial vehicle (UAV) swarms, characterized in that, include: The spatial module is used to model the multi-UAV cooperative collision avoidance problem as a distributed partially observable Markov decision process, defining the system state space, the local observation space of each UAV, and the joint action space. The calculation module is used to calculate the closest approach time and closest approach distance between any two drones based on their relative positions and relative speeds. When the nearest approach time is not negative and is less than the preset warning time window threshold, and the nearest approach distance is less than the preset safe distance threshold, it is determined that there is a potential conflict between the two machines, and a predictive conflict penalty item is triggered. The predictive conflict penalty terms of each UAV to its neighboring UAV within the communication range are aggregated to obtain the conflict cost signal of each UAV. The conflict cost signal is independent of the cooperative reward function and serves as the cost channel input for constrained reinforcement learning. The network module is used to train the action-evaluation policy network with shared parameters using a centralized training and decentralized execution approach. The optimization objective is to maximize the expected cumulative reward and ensure that the expected cumulative conflict cost does not exceed a preset constraint threshold. The module updates the policy network parameters and uses a proportional-integral-derivative control law to adaptively update the constraint multipliers. The output module is used to perform lightweight processing on the trained action network and deploy the lightweight action network to the onboard computing unit of each UAV. During the execution phase, each UAV independently runs the action network based only on its local observations and outputs collision avoidance control commands in real time.

Images