Unmanned aerial vehicle formation obstacle avoidance method and device based on distance attention reinforcement learning
Patent Information
- Application Number
- CN202610925999.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-25
- Publication Date
- 2026-08-18
AI Technical Summary
[0004]然而，现有基于强化学习的无人机编队避障方法大多仍依赖全局环境地图或预先构建的静态环境信息，难以适应低空空域中复杂多变的动态环境
[0052] The method described in this disclosure enables distributed autonomous decision-making based on local perception information of UAVs in unknown or dynamically changing environments, thereby improving the collaborative obstacle avoidance capability and formation stability of UAV swarms in complex environments. At the same time, by integrating multi-source environmental perception information and reinforcement learning decision-making mechanisms, it enables UAVs to collaboratively process static obstacles, dynamic obstacles, and formation status information, thereby improving the UAVs' path planning capability, obstacle avoidance flexibility, and mission execution safety in complex mixed obstacle environments. It also reduces the dependence on global environmental maps and centralized control, enhancing the robustness and generalization capability of the system.
Smart Images
Figure CN122593409A_ABST
Abstract
Description
Technical Field
[0001] This disclosure belongs to the field of UAV swarm cooperative control and reinforcement learning technology, specifically relating to a UAV formation obstacle avoidance method and device based on distance attention reinforcement learning. Background Technology
[0002] With the rapid development of the low-altitude economy, unmanned aerial vehicle (UAV) technology has gradually become an important component of low-altitude airspace applications. Compared to single-UAV systems, multi-UAV cooperative swarm systems, through the coordinated operation of multiple UAVs, can effectively improve mission execution efficiency, coverage, and system robustness, showing broad application prospects in fields such as agricultural inspection, logistics transportation, disaster relief, mountain surveying, and aerial patrol. During multi-UAV cooperative operations, UAVs not only need to complete target navigation tasks but also need to maintain a stable formation structure in complex environments and avoid static and dynamic obstacles in real time. Therefore, how to achieve UAV swarm control and safe obstacle avoidance in complex environments has become a key issue in current UAV swarm research.
[0003] Existing UAV swarm control methods mainly include centralized and distributed control methods. Centralized control methods typically employ a leader-follower structure or a virtual structure, achieving coordinated swarm control through unified planning of UAV flight trajectories. While this approach can guarantee swarm consistency to a certain extent, its strong dependence on the central node makes it prone to insufficient maneuverability and poor obstacle avoidance flexibility in complex dynamic environments. When numerous dynamic obstacles exist, centralized control struggles to respond promptly to local environmental changes, impacting system stability and safety. In contrast, distributed control methods allow each UAV to make independent decisions based on local observation information, exhibiting better flexibility and robustness, and improving the adaptability of UAV swarms in complex environments. However, in distributed obstacle avoidance, because each UAV makes independent decisions based on local information, behavioral deviations can easily arise between individuals, leading to significant disturbances in the swarm structure during obstacle avoidance and difficulty in quickly restoring the predetermined formation after obstacle avoidance. Therefore, it is difficult to balance obstacle avoidance flexibility and swarm consistency. In recent years, with the development of reinforcement learning technology, multi-agent reinforcement learning methods have been widely applied in UAV swarm control and cooperative obstacle avoidance. Reinforcement learning methods can autonomously learn control strategies through environmental interaction, exhibiting good environmental adaptability and policy generalization ability in complex environments. Some studies have improved the obstacle avoidance ability and cooperative control performance of UAV formations by introducing deep reinforcement learning, potential field reward mechanisms, and hierarchical training methods.
[0004] However, most existing reinforcement learning-based UAV formation obstacle avoidance methods still rely on global environment maps or pre-built static environment information, making it difficult to adapt to the complex and ever-changing dynamic environment in low-altitude airspace. When moving or unknown obstacles are present, non-real-time map update mechanisms can easily cause UAVs to fail to adjust their obstacle avoidance strategies in a timely manner, thus increasing the risk of collisions. Furthermore, in multi-UAV distributed cooperative scenarios, existing methods have limited ability to utilize multi-source information such as formation state, dynamic obstacles, and static obstacles, making it difficult to effectively coordinate the relationship between UAV formation maintenance and dynamic obstacle avoidance. This results in insufficient stability, consistency, and environmental adaptability of the system in complex environments.
[0005] Therefore, there is an urgent need to develop a drone formation obstacle avoidance method that is highly adaptable to the environment, has a fast dynamic obstacle response capability, and excellent formation consistency. Summary of the Invention
[0006] The purpose of this disclosure is to provide a method and apparatus for drone formation obstacle avoidance based on distance attention reinforcement learning, aiming to solve at least one technical problem existing in the prior art.
[0007] The technical solution disclosed herein is:
[0008] A method for UAV formation obstacle avoidance based on distance attention reinforcement learning is proposed. The UAVs rely on local perception to achieve distributed collaborative decision-making, and combine distance attention mechanisms with multi-agent reinforcement learning to maintain formation and achieve dynamic obstacle avoidance. Specifically, the method includes the following steps:
[0009] The drone collects multi-source state information through onboard sensors and uses local spatial projection to process static obstacle point cloud data to construct an environmental state space.
[0010] Features are extracted from any of the aforementioned state information, and attention weights for any feature are obtained by combining distance and formation error and then normalized to complete adaptive feature fusion.
[0011] The reference formation speed is obtained based on the formation consistency control algorithm. At the same time, the obstacle avoidance navigation speed is output by the reinforcement learning policy network according to the fusion environment characteristics. The two types of speeds are weighted and fused to generate the final flight control command of the UAV and execute it.
[0012] A multi-objective reward function that takes into account navigation, obstacle avoidance, formation maintenance, and control smoothness is constructed. A multi-agent proximal policy optimization algorithm is adopted to update the decision policy, thereby realizing dynamic obstacle avoidance and formation maintenance of UAV formations.
[0013] The UAV collects multi-source state information through onboard sensors, processes static obstacle point cloud data using local spatial projection, and constructs an environmental state space, including:
[0014] Real-time acquisition of UAV status information to construct a global state space;
[0015] For static obstacle point cloud data output by lidar, the local spatial projection method is used to complete the data transformation, and obtain a static obstacle state representation that can be used for network input;
[0016] The UAV status information includes one or more of the following: the UAV itself, the target point, dynamic obstacles, static obstacles, and formation status information.
[0017] The static obstacle point cloud data output by the lidar is transformed using a local spatial projection method to obtain a static obstacle state representation that can be used for network input, including:
[0018] Centered on the current position of the UAV, angular regions are divided along the horizontal and vertical directions. The average distance of the point cloud in each region is statistically analyzed to generate projection features in the corresponding directions, so that any vertical angle of the UAV... A set of horizontal spatial projection feature vectors is used, and the value of the obstacle-free area is the sum of the maximum detection distance and the preset offset. The projection features of each vertical angle are stacked to form a static obstacle state.
[0019] The drone at any vertical angle A corresponding set of horizontal spatial projection feature vectors is represented as:
[0020] ;in, Indicates the number of horizontally divided regions. This represents the statistical value of the distance from the i-th obstacle point cloud in the corresponding direction to the center of the UAV;
[0021] The static obstacle state Represented as:
[0022] ;
[0023] in, Indicates the number of vertical angle divisions; This is the horizontal spatial projection feature vector corresponding to the first dividing angle in the vertical dimension; The horizontal spatial projection feature vector corresponding to the angle at the end of the vertical dimension.
[0024] The state information is used to extract features, and the attention weight of any feature is obtained by combining distance and formation error and then normalized to complete the adaptive feature fusion, including:
[0025] The formation state, dynamic obstacle state, and static obstacle state corresponding to any of the aforementioned UAVs are respectively fed into a convolutional neural network to extract deep features. Based on the formation error and the relative distance of the obstacles, the attention weights for the three categories of formation, dynamic obstacles, and static obstacles are calculated respectively. After normalization processing, the extracted features are adaptively weighted and fused to obtain the fused environmental features.
[0026] The attention weight vector is represented as follows: ;
[0027] in, Assign attention weights to the formation; For dynamic obstacle attention weights; Attention weights for static obstacles;
[0028] Integrating environmental characteristics Represented as:
[0029] ;
[0030] in, The normalized formation attention weights; This refers to the formation state characteristics; This represents the normalized dynamic obstacle attention weights; Features of dynamic obstacles; The normalized static obstacle attention weights; This refers to the characteristics of a static obstacle.
[0031] The reference formation speed is obtained based on the formation consistency control algorithm. Simultaneously, a reinforcement learning policy network outputs obstacle avoidance and navigation speeds based on the fused environmental features. The two types of speeds are weighted and fused to generate and execute the final flight control commands for the UAV, including:
[0032] The control input for a drone is defined as follows: ;
[0033] in, Indicates the reference formation speed; Indicates the obstacle avoidance navigation speed; This represents the formation control weight coefficient; This represents the obstacle avoidance navigation control weight coefficient;
[0034] The output action space of a reinforcement learning policy network is defined as: ;
[0035] in, Indicates the normalized obstacle avoidance navigation speed;
[0036] Map the normalized obstacle avoidance navigation speed to the actual navigation control speed:
[0037] ;
[0038] in, This indicates the maximum navigation speed of the drone.
[0039] The construction of a multi-objective reward function that takes into account navigation, obstacle avoidance, formation maintenance, and control smoothness includes:
[0040] The multi-objective reward function is defined as follows:
[0041] ;
[0042] in, Indicates a reward for navigation to the target destination; Indicates a safety reward for static obstacles; Indicates a reward for dynamic obstacle avoidance; This indicates that the formation will maintain its reward; This indicates that the reward is controlled and smoothed out. These represent the weighting coefficients of the corresponding rewards.
[0043] The method of updating the decision strategy using a multi-agent proximal policy optimization algorithm to achieve dynamic obstacle avoidance and formation maintenance of UAV formations includes:
[0044] The system collects the state, actions, rewards, and next-moment state information of any UAV to form an experience sample, and uses the policy network and value network to update the UAV cooperative control strategy.
[0045] The policy update process is optimized using a dominance function, and the policy update magnitude is limited by truncation probability ratio.
[0046] The strategy optimization objective is defined as:
[0047] ;
[0048] in, This represents the ratio of the probabilities of the new and old strategies; Indicates the policy truncation threshold; Indicates time-based Empirical expectation operator for sampling trajectories; Indicates time The corresponding estimated advantage function; This represents a interval cutoff function, used to restrict the input value to a range of... Within the range.
[0049] A drone formation obstacle avoidance device based on distance attention reinforcement learning includes: a storage medium and a processing unit;
[0050] The storage medium is used to store the computer program described above. The processing unit exchanges data with the storage medium to execute the computer program and perform the steps of the above method when the UAV formation avoids obstacles.
[0051] The beneficial effects of this disclosure include at least the following:
[0052] The method described in this disclosure enables distributed autonomous decision-making based on local perception information of UAVs in unknown or dynamically changing environments, thereby improving the collaborative obstacle avoidance capability and formation stability of UAV swarms in complex environments. At the same time, by integrating multi-source environmental perception information and reinforcement learning decision-making mechanisms, it enables UAVs to collaboratively process static obstacles, dynamic obstacles, and formation status information, thereby improving the UAVs' path planning capability, obstacle avoidance flexibility, and mission execution safety in complex mixed obstacle environments. It also reduces the dependence on global environmental maps and centralized control, enhancing the robustness and generalization capability of the system. Attached Figure Description
[0053] Figure 1 This is a flowchart of the drone formation obstacle avoidance method based on distance attention reinforcement learning as described in this disclosure;
[0054] Figure 2 This is a schematic diagram of a dynamic obstacle avoidance scenario for drone formations in the Gazebo simulation environment;
[0055] Figure 3 This is a schematic diagram of the drone formation obstacle avoidance flight trajectory in the RViz environment;
[0056] Figure 4 This is a graph showing the variation of consistency error in drone swarm formation;
[0057] Figure 5 It is a convergence curve of reward during multi-agent reinforcement learning training. Detailed Implementation
[0058] The following describes specific embodiments of this disclosure to enable those skilled in the art to understand it. However, it should be understood that this disclosure is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of this disclosure as defined and determined by the appended claims. All inventions utilizing the concept of this disclosure are protected. Specific Implementation Example 1:
[0060] This disclosure provides an embodiment:
[0061] This embodiment provides a drone formation obstacle avoidance method based on distance attention reinforcement learning, applicable to multi-drone cooperative formation flight and autonomous obstacle avoidance tasks in complex dynamic environments. Figure 1 As shown, this embodiment mainly includes three parts: environmental state perception, collaborative decision control, and fusion control output. The overall idea is as follows: First, the UAV acquires surrounding environmental state information through onboard sensors, including the UAV's own state, formation state information, dynamic obstacle information, and static obstacle information. Then, the environmental state information is input into the collaborative decision module based on a distance attention mechanism. The attention mechanism is used to perform feature weighting on different observation information, and corresponding control quantities are generated by combining reinforcement learning strategies. Finally, the formation control quantity and obstacle avoidance navigation control quantity are fused to output the final control command of the UAV, realizing UAV collaborative formation obstacle avoidance in complex dynamic environments.
[0062] The specific implementation steps of this embodiment are as follows:
[0063] Step 1: In the environmental state perception phase, the UAV acquires surrounding environmental state information through onboard sensors and constructs the state space for the current moment. At any given moment... Define the current state of each drone as .in, This indicates the drone's own status information. Indicates the target point status information. Represents dynamic obstacle status information. Represents the state information of static obstacles. This indicates the formation status information. By constructing a local observation space for the UAV using the aforementioned multi-source environmental status information, an environmental perception foundation is provided for subsequent collaborative decision-making and obstacle avoidance control.
[0064] Specifically, the state space defined in this embodiment The following is the drone's own status. Defined as: ;in, Indicates the current location of the drone. This indicates the drone's current speed. Target point status. The term used to characterize the relative direction and distance between the UAV and the target point is defined as: ;
[0065] in, Indicates the location of the target point. This indicates the relative distance between the drone's current position and the target point. Dynamic obstacle status. Used to describe the relative motion relationship between dynamic obstacles and drones, defined as: ;
[0066] in, This represents the total number of dynamic obstacles. Let be the dimension of the state vector for a single obstacle. The state vector for each obstacle is defined as: .in, Indicates the first The center position of a dynamic obstacle For the first The speed of the obstacle.
[0067] Static obstacle status The system is constructed using LiDAR point cloud information. Since static obstacles are represented in discrete point cloud form, they cannot be directly used as input for reinforcement learning decision networks. Therefore, a static obstacle representation method based on local spatial projection is used to process the point cloud data.
[0068] Specifically, taking the drone's current position as the center, the space within a 360° radius around the drone is divided into multiple horizontal regions. The average distance from the obstacle point cloud within each region to the drone's center position is then calculated, thereby constructing horizontal spatial projection features in the corresponding direction. In the vertical direction, the same method is used to perform spatial projection processing on point cloud data within different vertical angle ranges, ensuring that each vertical angle... Corresponding to a set of horizontal spatial projection feature vectors: .in, Indicates the number of horizontally divided regions. This represents the statistical value of the distance from the obstacle point cloud to the center of the UAV in the corresponding direction. When no obstacle is detected in the corresponding direction, the corresponding projected distance is assigned as the sum of the maximum detection distance and a preset offset to indicate that there is no obstacle in the current direction. Finally, the spatial projection features at different vertical angles are stacked to form a static obstacle state representation: ;in, This indicates the number of angle divisions in the vertical direction.
[0069] For formation status The information employs a formation state representation method based on consistency control. By constructing relative positional constraints between UAVs, the formation structure of the UAV swarm is maintained. The global formation error of the UAV system is defined as: ;
[0070] in, Indicates the current location and status of the drone cluster. Indicates the desired formation position. This represents the Laplace matrix corresponding to the UAV communication topology. Represents a three-dimensional identity matrix. This represents the Kronecker product operation.
[0071] To maintain drone formation consistency, the drone formation control input is defined as:
[0072] ;
[0073] in, Indicates drone The set of neighboring nodes, Indicates the communication weight between drones. This indicates the formation consistency control gain.
[0074] Furthermore, the consistency control output is used as a reference formation speed: And together with the formation error, the formation state information is constructed: .in, Indicates the reference control speed of the drone formation. This indicates the deviation between the current position of the drone and the desired formation position.
[0075] Step 2: In the collaborative decision-making stage, the environmental state information constructed in Step 1 is input into the collaborative decision-making network based on the distance attention mechanism, and different environmental observation features are dynamically weighted.
[0076] First, the formation status information Dynamic obstacle status information and static obstacle status information Each input is processed into a convolutional neural network for feature extraction, yielding the corresponding high-dimensional feature representations: .in, Indicates the formation status characteristics, Indicates dynamic obstacle characteristics, This indicates the characteristics of a static obstacle.
[0077] Furthermore, corresponding attention weights are constructed based on formation error, dynamic obstacle distance, and static obstacle distance.
[0078] The formation attention weight is defined as: ;
[0079] in, Indicates the first Formation error of drones This represents the formation error weighting coefficient.
[0080] The dynamic obstacle attention weight is defined as: ;
[0081] in, Indicates the current location of the drone. Indicates the first The location of a dynamic obstacle. Indicates the maximum sensing distance. This represents the dynamic obstacle attenuation coefficient.
[0082] The static obstacle attention weight is defined as:
[0083] ;
[0084] in, This indicates the minimum distance between the current drone and a static obstacle. Represents the static obstacle attenuation coefficient. Indicates the effective sensing range. This represents a preset constant.
[0085] The three types of attention weights are used to construct the attention vector: And normalization is performed using the Softmax function: ;
[0086] in, This represents the normalized attention weights.
[0087] Finally, the normalized attention weights are applied to the environmental feature fusion to obtain the fused environmental feature representation: ;
[0088] in, This represents the fused environmental feature vector.
[0089] Step 3: In the fusion control output stage, in order to balance the consistency of UAV swarm formation and local dynamic obstacle avoidance capabilities, a fusion control method combining formation control and obstacle avoidance navigation is adopted.
[0090] The final control input of the UAV is defined as: ;
[0091] in, This represents the reference formation speed generated in step 1. Indicates obstacle avoidance navigation speed. This represents the formation control weighting coefficient. This represents the obstacle avoidance navigation control weight coefficient.
[0092] Furthermore, the output action space of the reinforcement learning policy network is defined as: ;
[0093] in, This represents the normalized obstacle avoidance navigation speed.
[0094] Then, the normalized navigation speed is mapped to the actual navigation control speed: ;
[0095] in, This indicates the maximum navigation speed of the drone.
[0096] Step 4: During reinforcement learning training, a multi-objective joint reward function is constructed to balance target navigation, obstacle avoidance, formation maintenance, and control smoothness. For the... The reward function for a drone is defined as: ;
[0097] in, Indicates a reward for navigation to the target destination. Indicates a safety reward for static obstacles. This indicates a reward for dynamic obstacle avoidance. This indicates that the formation will maintain its reward. This indicates that the reward is controlled to be smooth. These represent the corresponding reward weight coefficients.
[0098] Target navigation rewards Defined as: ;
[0099] in, Indicates the target location. Indicates the current location of the drone. This indicates the obstacle avoidance navigation speed.
[0100] Static obstacle safety reward Defined as: ;
[0101] in, This represents the Sigmoid function. Indicates a safe distance. It is a constant.
[0102] Dynamic obstacle avoidance reward Defined as: ;
[0103] in, Indicates the first The location of a dynamic obstacle.
[0104] Formation maintains rewards Defined as: ;
[0105] in, This indicates the current drone formation error. This is the attenuation coefficient.
[0106] Controlling smooth rewards Defined as: ;
[0107] in, , This represents the corresponding weighting coefficient.
[0108] Finally, the rewards of all drones are summed to obtain the overall reward function for the drone swarm, expressed as: .
[0109] Step 5: In the policy training phase, the multi-agent proximal policy optimization algorithm (MAPPO) is used to train the UAV cooperative decision-making strategy. Each UAV agent executes actions based on the current environmental state, interacts with the environment, and obtains corresponding reward values and state information for the next time step.
[0110] Subsequently, the current state, actions, rewards, and the state at the next moment are used to form an experience sample, and the cooperative control strategy for the UAV is updated using a policy network and a value network. The policy network outputs the UAV action control strategy, while the value network evaluates the value of the current state.
[0111] Furthermore, the advantage function is used to optimize the policy update process, and the policy update magnitude is limited by the truncation probability ratio. The policy optimization objective is defined as follows:
[0112] ;
[0113] in, This represents the ratio of the probabilities of the new and old strategies. Represents the dominance function. This indicates the policy truncation threshold.
[0114] Verification process:
[0115] In the simulation environment, the UAV employs a multi-rotor dynamics model and combines it with airborne sensors such as lidar to acquire information about the surrounding environment. To improve the realism of the simulation environment, communication delays, sensor noise, and external random disturbances are introduced into the system to simulate a complex dynamic low-altitude environment.
[0116] like Figure 2 As shown, the Gazebo simulation environment contains 30 randomly distributed static obstacles and 5 dynamically moving obstacles. Multiple drones perform cooperative formation flight and dynamic obstacle avoidance tasks in an unknown environment. Figure 2 The image shows the formation changes of the drone swarm at different time points and the movement trajectory of dynamic obstacles.
[0117] During the experiment, the UAV completed environmental state perception, attention feature weighting, fusion control output, and autonomous obstacle avoidance decision-making based on the distance attention reinforcement learning collaborative control method provided in this embodiment. Figure 3The results show the overall flight trajectory of the UAV swarm in the RViz environment. It can be seen that the UAVs can complete stable formation flight in complex obstacle environments and achieve autonomous obstacle avoidance of dynamic obstacles. Figure 4 The results show the variation of drone swarm consistency error, including curves illustrating the changes in position and velocity errors. Experimental results show that during dynamic obstacle avoidance, each drone gradually recovers to a stable formation state, and the position and velocity errors eventually converge, indicating that the method provided in this embodiment can effectively maintain the consistency and motion stability of the drone swarm formation. Furthermore, Figure 5 This is the reward change curve during the training process of the method described in this embodiment. As the number of training iterations increases, the reward value gradually increases and eventually stabilizes, indicating that the distance attention reinforcement learning collaborative control method provided in this embodiment can achieve effective policy learning and has good training stability and convergence performance.
[0118] In summary, the method provided in this embodiment first processes LiDAR point cloud data using a local spatial projection method to construct static obstacle features, achieving obstacle state perception in complex 3D environments and adapting to reinforcement learning input. Then, relying on a distance attention mechanism, it assigns weights to the states of formation, dynamic obstacles, and static obstacles to complete adaptive collaborative obstacle avoidance decisions in complex dynamic environments. Finally, it integrates consistent formation control and reinforcement learning-based obstacle avoidance navigation control to achieve dynamic obstacle avoidance while ensuring the stability of the UAV swarm formation. Furthermore, the verification in this step clearly demonstrates that this embodiment can achieve stable formation flight and dynamic autonomous obstacle avoidance of UAV swarms in complex dynamic environments, verifying the effectiveness and robustness of the distance attention-based reinforcement learning-based UAV formation obstacle avoidance method provided in this embodiment in complex low-altitude environments. Specific Implementation Example 2:
[0120] This disclosure also provides an embodiment:
[0121] An electronic device includes: a storage medium and a processing unit; wherein the storage medium is used to store the computer program as described above, and the processing unit exchanges data with the storage medium to execute the computer program during drone formation obstacle avoidance, performing the steps of the drone formation obstacle avoidance method based on distance attention reinforcement learning as described in Specific Embodiment 1. Specific Implementation Example 3:
[0123] A computer-readable storage medium storing a computer program; when the computer program is run, it performs the steps of the method as described in Specific Embodiment 1.
[0124] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, etc., or any suitable combination thereof.
[0125] The above disclosure only discloses a few specific implementation scenarios. However, this disclosure is not limited to these. Any variations that can be conceived by those skilled in the art should fall within the protection scope of this disclosure.
Claims
1. A drone formation obstacle avoidance method based on distance attention reinforcement learning, characterized in that, The drone relies on local perception to achieve distributed collaborative decision-making, and combines distance attention mechanisms and multi-agent reinforcement learning to complete formation maintenance and dynamic obstacle avoidance. Specifically, it includes the following steps: The drone collects multi-source state information through onboard sensors and uses local spatial projection to process static obstacle point cloud data to construct an environmental state space. Features are extracted from any of the aforementioned state information, and attention weights for any feature are obtained by combining distance and formation error and then normalized to complete adaptive feature fusion. The reference formation speed is obtained based on the formation consistency control algorithm. At the same time, the obstacle avoidance navigation speed is output by the reinforcement learning policy network according to the fusion environment characteristics. The two types of speeds are weighted and fused to generate the final flight control command of the UAV and execute it. A multi-objective reward function that takes into account navigation, obstacle avoidance, formation maintenance, and control smoothness is constructed. A multi-agent proximal policy optimization algorithm is adopted to update the decision policy, thereby realizing dynamic obstacle avoidance and formation maintenance of UAV formations.
2. The UAV formation obstacle avoidance method based on distance attention reinforcement learning according to claim 1, characterized in that, The UAV collects multi-source state information through onboard sensors, processes static obstacle point cloud data using local spatial projection, and constructs an environmental state space, including: Real-time acquisition of UAV status information to construct a global state space; For static obstacle point cloud data output by lidar, the local spatial projection method is used to complete the data transformation, and obtain a static obstacle state representation that can be used for network input; The UAV status information includes one or more of the following: the UAV itself, the target point, dynamic obstacles, static obstacles, and formation status information.
3. The UAV formation obstacle avoidance method based on distance attention reinforcement learning according to claim 2, characterized in that, The static obstacle point cloud data output by the lidar is transformed using a local spatial projection method to obtain a static obstacle state representation that can be used for network input, including: Centered on the current position of the UAV, angular regions are divided along the horizontal and vertical directions. The average distance of the point cloud in each region is statistically analyzed to generate projection features in the corresponding directions, so that any vertical angle of the UAV... A set of horizontal spatial projection feature vectors is used, and the value of the obstacle-free area is the sum of the maximum detection distance and the preset offset. The projection features of each vertical angle are stacked to form a static obstacle state. The drone at any vertical angle A corresponding set of horizontal spatial projection feature vectors is represented as: ;in, Indicates the number of horizontally divided regions. This represents the statistical value of the distance from the i-th obstacle point cloud in the corresponding direction to the center of the UAV; The static obstacle state Represented as: ； in, Indicates the number of vertical angle divisions; This is the horizontal spatial projection feature vector corresponding to the first dividing angle in the vertical dimension; The horizontal spatial projection feature vector corresponding to the angle at the end of the vertical dimension.
4. The UAV formation obstacle avoidance method based on distance attention reinforcement learning according to claim 1, characterized in that, The state information is used to extract features, and the attention weight of any feature is obtained by combining distance and formation error and then normalized to complete the adaptive feature fusion, including: The formation state, dynamic obstacle state, and static obstacle state corresponding to any of the aforementioned UAVs are respectively fed into a convolutional neural network to extract deep features. Based on the formation error and the relative distance of the obstacles, the attention weights for the three categories of formation, dynamic obstacles, and static obstacles are calculated respectively. After normalization processing, the extracted features are adaptively weighted and fused to obtain the fused environmental features.
5. The UAV formation obstacle avoidance method based on distance attention reinforcement learning according to claim 4, characterized in that, include: The attention weight vector is represented as follows: ; in, Assign attention weights to the formation; For dynamic obstacle attention weights; Attention weights for static obstacles; Integrating environmental characteristics Represented as: ； in, The normalized formation attention weights; This refers to the formation state characteristics; This represents the normalized dynamic obstacle attention weights; Features of dynamic obstacles; The normalized static obstacle attention weights; This refers to the characteristics of a static obstacle.
6. The UAV formation obstacle avoidance method based on distance attention reinforcement learning according to claim 1, characterized in that, The reference formation speed is obtained based on the formation consistency control algorithm. Simultaneously, a reinforcement learning policy network outputs obstacle avoidance and navigation speeds based on the fused environmental features. The two types of speeds are weighted and fused to generate and execute the final flight control commands for the UAV, including: The control input for a drone is defined as follows: ; in, Indicates the reference formation speed; Indicates the obstacle avoidance navigation speed; This represents the formation control weight coefficient; This represents the obstacle avoidance navigation control weight coefficient; The output action space of a reinforcement learning policy network is defined as: ; in, Indicates the normalized obstacle avoidance navigation speed; Map the normalized obstacle avoidance navigation speed to the actual navigation control speed: ； in, This indicates the maximum navigation speed of the drone.
7. The UAV formation obstacle avoidance method based on distance attention reinforcement learning according to claim 1, characterized in that, The construction of a multi-objective reward function that takes into account navigation, obstacle avoidance, formation maintenance, and control smoothness includes: The multi-objective reward function is defined as follows: ； in, Indicates a reward for navigation to the target destination; Indicates a safety reward for static obstacles; Indicates a reward for dynamic obstacle avoidance; This indicates that the formation will maintain its reward; This indicates that the reward is controlled and smoothed out. These represent the weighting coefficients of the corresponding rewards.
8. The UAV formation obstacle avoidance method based on distance attention reinforcement learning according to claim 1, characterized in that, The method of updating the decision strategy using a multi-agent proximal policy optimization algorithm to achieve dynamic obstacle avoidance and formation maintenance of UAV formations includes: The system collects the state, actions, rewards, and next-moment state information of any UAV to form an experience sample, and uses the policy network and value network to update the UAV cooperative control strategy. The policy update process is optimized using a dominance function, and the policy update magnitude is limited by truncation probability ratio.
9. The UAV formation obstacle avoidance method based on distance attention reinforcement learning according to claim 8, characterized in that: The strategy optimization objective is defined as: ； in, This represents the ratio of the probabilities of the new and old strategies; Indicates the policy truncation threshold; Indicates time-based Empirical expectation operator for sampling trajectories; Indicates time The corresponding estimated advantage function; This represents a interval cutoff function, used to restrict the input value to a range of... Within the range.
10. A drone formation obstacle avoidance device based on distance attention reinforcement learning, characterized in that, include: Storage media and processing units; The storage medium is used to store a computer program, and the processing unit exchanges data with the storage medium to execute the computer program during drone formation obstacle avoidance, performing the steps of the method as described in any one of claims 1-9.