Multi-unmanned aerial vehicle path planning obstacle avoidance method and system based on Rainbow DQN algorithm

Abstract

The invention provides a multi-unmanned aerial vehicle path planning obstacle avoidance method and system based on a Rainbow DQN algorithm, and the technical scheme adopted by the invention is as follows: firstly, constructing a multi-unmanned aerial vehicle system model, and initializing a training data set by using a trajectory fragment generated by a reciprocal collision avoidance algorithm and a multi-step time sequence difference target value; in the training process, priority sampling is adopted to carry out importance weighting on samples, and an attention mechanism is introduced to carry out enhancement processing on state features; a local value network of each unmanned aerial vehicle is constructed in combination with an improved depth Q network, local values are fused into a global value through a mixed value network, and finally joint training is performed based on a global loss function of a corrected weight, so that cooperative path planning and obstacle avoidance of multiple unmanned aerial vehicles are realized. The technical problems of low cooperative obstacle avoidance efficiency, large value estimation deviation and unstable training of multiple unmanned aerial vehicles in a complex dynamic environment in the prior art can be solved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) path planning technology, specifically to a multi-UAV path planning and obstacle avoidance method and system based on the Rainbow DQN algorithm. Background Technology

[0002] With the rapid development of drone technology, multi-drone collaborative systems are increasingly being used in agricultural plant protection, disaster relief, logistics distribution, and military reconnaissance. In these complex mission scenarios, multiple drones need to meet the requirements of collaborative operation while avoiding static terrain and dynamic obstacles in real time, and preventing collisions within the drone swarm. Therefore, efficient and robust path planning and obstacle avoidance algorithms are crucial for achieving autonomous multi-drone collaboration.

[0003] Existing multi-UAV path planning methods mainly include traditional planning algorithms (such as the artificial potential field method and ORCA algorithm) and methods based on deep reinforcement learning. While traditional algorithms have lower computational cost, they are prone to getting trapped in local optima when facing complex dynamic environments, leading to path oscillations or deadlocks, and they struggle to handle complex collaborative relationships between multiple UAVs. In recent years, methods based on deep reinforcement learning (such as DQN and MADDPG) have attracted considerable attention due to their powerful perception and decision-making capabilities, but they still face many challenges in practical applications. First, multi-UAV systems are a typical high-dimensional state space and unsteady environment. Traditional independent reinforcement learning methods ignore the mutual influence between agents, resulting in unstable training and difficulty in convergence. Second, existing reinforcement learning algorithms usually adopt random exploration strategies, which have problems such as high blindness, high trial and error costs, and slow convergence speed in the early stage of training. Moreover, they have low utilization of sparse but critical samples such as "collision edges" and "near-distance obstacle avoidance", resulting in insufficient obstacle avoidance success rate.

[0004] Furthermore, during dense swarm flights, UAVs need to process a large amount of neighbor information. Existing network structures often average all perceived information, making it impossible to automatically focus on the key targets that pose the greatest threat to themselves from redundant information, resulting in decision-making delays. Finally, the basic DQN algorithm suffers from value overestimation, and a single scalar value cannot accurately reflect the random distribution of the environment, limiting the superiority of the strategy.

[0005] Therefore, designing a path planning method that can ensure individual obstacle avoidance safety, achieve efficient group collaboration, and simultaneously possess high training efficiency and value estimation accuracy is a problem that urgently needs to be solved. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a multi-UAV path planning and obstacle avoidance method based on the Rainbow DQN algorithm, in order to solve the technical problems of low efficiency, large value estimation bias, and unstable training of multi-UAV cooperative obstacle avoidance in complex dynamic environments.

[0007] The technical solution adopted in this invention is as follows: First, a multi-UAV system model is constructed, and the training dataset is initialized using trajectory segments generated by the reciprocal collision avoidance algorithm and multi-step temporal differential target values. During the training process, priority sampling is used to weight the importance of the samples, and an attention mechanism is introduced to enhance the state features. The local value network of each UAV is constructed by combining an improved deep Q-network, and the local values ​​are merged into a global value through a hybrid value network. Finally, joint training is performed based on a global loss function with corrected weights to achieve collaborative path planning and obstacle avoidance of multiple UAVs.

[0008] In the first feasible approach, a multi-UAV path planning and obstacle avoidance method based on the Rainbow DQN algorithm is provided, including: Step S1: Construct a system model that includes information on multiple drones and the environment; Step S2: Define the local state vector and global state vector of the UAV; Step S3: Generate collision-free operating trajectories for multiple UAVs, calculate the target value of trajectory segments using the multi-step temporal difference formula, and store the training samples containing local state vectors, global state vectors, and target values ​​into the training dataset. Step S4: Sample the training samples from the training dataset, and perform sample importance weighting and state feature enhancement on the training samples respectively to obtain the corrected weights and enhanced state vectors; Step S5: Construct a local value network for each UAV and output the local action value using the enhanced state vector; Step S6: Construct a hybrid value network, receive the local action values ​​of all UAVs, and fuse them with the global state vector to form a global value; Step S7: Calculate the global loss function using the corrected weights, and jointly train the local value network and the hybrid value network.

[0009] Furthermore, step S1 specifically includes the following steps: Establish a three-dimensional coordinate system as the environmental space, and define the spatial attributes of terrain, static obstacles, and dynamic obstacles; Define a collision avoidance zone for each drone. The collision avoidance zone is a spherical area with the drone itself as the origin and a preset collision avoidance threshold as the radius. Define the local perception range of each drone to obtain relative information between the drone and its surrounding environment.

[0010] Furthermore, step S2 specifically includes the following steps: The local state vector includes the UAV's own position, speed, flight azimuth angle, and the relative distance, relative speed, relative speed angle, and collision risk coefficient between the UAV and neighboring UAVs and obstacles within the local perception range. The global state vector includes the three-dimensional position coordinates of all UAVs in the environmental space and the distance information of each UAV from its target point; The formula for calculating the collision risk coefficient is as follows: in, Indicates the risk coefficient. Indicates relative distance. This indicates the angle between relative velocities.

[0011] Furthermore, step S3 specifically includes the following steps: Define the number of drones and the target point in the environmental space, call the reciprocal collision avoidance algorithm to generate collision-free running trajectories of multiple drones, and use the running trajectories as initial training samples; Define the step size n for multi-step temporal difference learning, and decompose the initial training sample into continuous trajectory segments containing n time steps; Construct an experience replay pool to store the continuous trajectory segments, and parse each trajectory segment into a state vector sequence within n time steps; The target value of the trajectory segment is calculated using a multi-step temporal difference formula, which is expressed as follows: in, This represents the target value at time step t. to Indicates from time step t to Instant rewards Indicates the discount factor. Indicates the target network's time steps state Value assessment; The local state vector, global state vector, and target value corresponding to the trajectory segment are used to form training samples and stored in the training dataset.

[0012] Furthermore, in step S4, the training samples are weighted according to their importance. The specific steps include: Construct an experience replay pool that supports priority, storing training samples and their corresponding temporal difference errors and priorities, wherein the temporal difference error of the initial training samples is set to a preset maximum value; The sampling priority of the samples is calculated using the time-series difference error, and batch samples are extracted according to the sampling priority; The formula for calculating the sampling priority is as follows: Where P(i) represents the sampling probability, Let represent the temporal difference error of the i-th training sample, and α represent the priority weight coefficient; Calculate the importance sampling weights of the batch samples, which will be used to correct the gradient update step size in subsequent steps; The formula for calculating the importance sampling weight is as follows: in, The importance sampling weight of the i-th sample is represented by N, where N represents the total sample capacity of the experience replay pool. This indicates the sample sampling probability or the normalized priority. The parameter representing the bias in balancing priority is This represents the largest weight value in the current batch.

[0013] Furthermore, in step S4, state feature enhancement is performed on the training samples, specifically including the following steps: The attention weights between the drone itself, neighboring drones, and obstacles are calculated using both distance-based exponential decay and collision risk coefficient-based weight decay. The attention weights are used to weight the environmental interaction information within the local perception range, and the weighted features are concatenated with the UAV's own motion state to obtain the enhanced state vector.

[0014] Furthermore, in step S5, the local value network adopts an improved deep Q-network architecture, including a main network and a target network with the same structure. Both the main network and the target network are configured with the following structure: A competitive network is used to split the output of the fully connected layer of the network into a state value branch V(s) and an advantage value branch A(s,a), and calculates the action value using the following formula: in, Indicates the value of an action. Represents the value of a state. This indicates the advantage value of action a when collected in state s. This represents the mean of the dominance values ​​for all actions. Noisy networks are used to introduce Gaussian noise parameters into fully connected layers, and the randomness of the network parameters enables an exploration strategy to replace the greedy strategy. Distributed value estimation is used to discretize the output of the action value into a probability distribution containing several value atoms; During training, the main network is used to select the optimal action in the current state, and the target network is used to evaluate the value distribution of the optimal action.

[0015] Furthermore, step S6 specifically includes; Construct a QMIX hybrid value network, which includes a hybrid layer for inputting the local action values ​​of each UAV and a super network for inputting the global state vector; Based on the global state vector through the hypernetwork Generate non-negative weights and bias terms ; The hybrid layer is used to fuse the local motion values ​​[Q1,Q2,...,Qn] of all UAVs into a global value, calculated as follows: in, Indicates overall value. Let s represent the local value of the i-th drone, and sglobal represent the global state.

[0016] Construct a global target value network with the same structure as the QMIX hybrid value network to calculate the global target value. The calculation formula is as follows: in, express, Indicates global reward. Indicates the discount factor. Indicates the step size in multi-step learning. This indicates that the global target value network is determined based on the next state. and optimal action combination Calculated evaluation value.

[0017] Furthermore, step S7 specifically includes: Construct a global loss function, and weight the prediction error of the global value according to the modified weights; By minimizing the global loss function, the parameters of both the local value network and the hybrid value network are updated simultaneously. The global loss function is expressed as: in, This indicates the number of samples in the training batch. Indicates the sample index. The importance sampling weight of the i-th sample is represented. This represents the mean square error function. Represents the global target value of the i-th sample. , Represents the global value of the i-th sample. .

[0018] In conjunction with the first feasible method, a second feasible method provides a multi-UAV path planning and obstacle avoidance system based on the Rainbow DQN algorithm, characterized by comprising: The environment modeling module is used to build a system model that includes information about multiple drones and the environment, and to define the local state vector and global state vector of the drones. The trajectory generation and sample initialization module is used to generate collision-free operating trajectories for multiple UAVs, calculate the target value of trajectory segments using multi-step temporal difference formulas, and store training samples containing local state vectors, global state vectors and target values ​​into the training dataset. The priority sampling and feature enhancement module is used to sample the training samples from the training dataset, and perform sample importance weighting and state feature enhancement on the training samples respectively to obtain the corrected weights and enhanced state vectors. The local value network construction module is used to construct a local value network for each UAV and output the local action value using the enhanced state vector; The global value hybrid training module is used to construct a hybrid value network, receive the local action values ​​of all UAVs, and fuse them with the global state vector to form a global value; it uses the corrected weights to calculate the global loss function and jointly trains the local value network and the hybrid value network.

[0019] As can be seen from the above technical solution, the beneficial technical effects of the present invention are as follows: 1. By initializing the training dataset with the generated trajectory and combining it with a priority experience replay mechanism that supports temporal difference errors, the problem of large blindness in the initial exploration and slow convergence speed of traditional reinforcement learning is solved, and the learning efficiency of key sparse samples such as collision edges is significantly improved.

[0020] 2. By using a self-attention mechanism to enhance the features of local perception information, the UAV can automatically focus on high-risk neighbors and obstacles. At the same time, by combining an improved deep Q-network architecture, the defect of value overestimation in the single DQN algorithm is effectively overcome, and the robustness of decision-making in complex dynamic environments is improved. Attached Figure Description

[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0022] Figure 1 This is a schematic diagram of the algorithm structure in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the Rainbow DQN algorithm combined with the QMIX network in Embodiment 1 of the present invention; Figure 3 This is a system structure diagram of Embodiment 2 of the present invention; Figure label: 21-A multi-UAV path planning and obstacle avoidance system based on Rainbow DQN algorithm; 22-Environment modeling module; 23-Trajectory generation and sample initialization module; 24-Priority sampling and feature enhancement module; 25-Local value network construction module; 26-Global value hybrid training module. Detailed Implementation

[0023] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0024] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0025] Example 1 This embodiment provides a multi-UAV path planning and obstacle avoidance method based on the Rainbow DQN algorithm. The working principle of Embodiment 1 is explained in detail below: The method flow of this embodiment includes: Step S1: Construct a system model that includes information on multiple drones and the environment; Step S2: Define the local state vector and global state vector of the UAV; Step S3: Generate collision-free operating trajectories for multiple UAVs, calculate the target value of trajectory segments using the multi-step temporal difference formula, and store the training samples containing local state vectors, global state vectors, and target values ​​into the training dataset. Step S4: Sample the training samples from the training dataset, and perform sample importance weighting and state feature enhancement on the training samples respectively to obtain the corrected weights and enhanced state vectors; Step S5: Construct a local value network for each UAV and output the local action value using the enhanced state vector; Step S6: Construct a Mixing Network to receive the local action values ​​of all UAVs and fuse them with the global state vector into a global value; Step S7: Calculate the global loss function using the corrected weights, and jointly train the local value network and the hybrid value network.

[0026] In this embodiment, step S1 further includes the following specific steps: Establish a three-dimensional coordinate system as the environmental space, with the x, y, and z axes corresponding to the east-west, north-south, and altitude directions, respectively.

[0027] Define the spatial attributes of terrain, static obstacles, and dynamic obstacles. Set the radius to 0.5-2m according to the UAV specifications (divided into micro / small). Quantify terrain elevation data (accuracy 0.1m), static obstacles (including the three-dimensional coordinates and dimensions of buildings and towers), and dynamic obstacles (including the movement trajectory, speed range, and acceleration range of pedestrians and vehicles).

[0028] Define a collision avoidance zone for each drone. The collision avoidance zone is a spherical area with the drone itself as the origin and a preset collision avoidance threshold as the radius. Define the local perception range of each drone to obtain relative information between the drone and its surrounding environment.

[0029] In this embodiment, step S2 further includes the following specific steps: The local state vector includes the UAV's own position, speed, flight azimuth angle, and the relative distance, relative speed, relative speed angle, and collision risk coefficient between the UAV and neighboring UAVs and obstacles within the local perception range. The global state vector includes the three-dimensional position coordinates of all UAVs in the environmental space and the distance information of each UAV from its target point; At each time step t (0.1s), a collision risk coefficient is calculated. A higher collision risk coefficient indicates a higher risk. The formula for calculating the collision risk coefficient is as follows: in, Indicates the risk coefficient. Indicates relative distance. This indicates the angle between relative velocities.

[0030] In this embodiment, step S3 further includes the following steps: Based on the task requirements, set the number of drones and target points in the environment, call the reciprocal collision avoidance algorithm (ORCA) to generate collision-free running trajectories of multiple drones (obstacle avoidance success rate ≥95%), and use the running trajectories as initial training samples. Set the multi-step temporal difference learning step size n=4, and decompose the initial training sample into continuous trajectory segments containing 4 time steps; An experience replay pool supporting multi-step sequence storage is constructed to store the continuous trajectory segments. Each trajectory segment is parsed into a state vector sequence within 4 time steps, specifically including a state sequence, an action sequence, and a reward sequence. The reward sequence includes an obstacle avoidance success reward, a shortest path reward, and a collision penalty.

[0031] The target value of the trajectory segment is calculated using a multi-step temporal difference formula, thus completing the initialization of the "state-value" pair.

[0032] The multi-step timing difference formula is expressed as follows: in, This represents the target value at time step t. to Indicates from time step t to Instant rewards Indicates the discount factor. Indicates the target network's time steps state In this embodiment, the discount factor is used for value assessment. .

[0033] The local state vector, global state vector, and target value corresponding to the trajectory segment are used to form training samples and stored in the training dataset.

[0034] In this embodiment, further, in step S4, the training samples are weighted according to their importance, specifically including the following steps: Construct an experience replay pool that supports priority, storing training samples and their corresponding temporal difference errors and priorities. The temporal difference error of the initial training samples is set to a preset maximum value of 1 to ensure that the initial high-quality samples are learned first. The sampling priority of the samples is calculated using the time difference error, and batch samples are extracted according to the sampling priority to increase the sampling probability of large TD (time difference) error samples such as "collision edge" and "near distance obstacle avoidance".

[0035] The formula for calculating the sampling priority is as follows: Where P(i) represents the sampling probability, This represents the temporal difference error of the i-th training sample; Introducing IS weight correction (a parameter to balance priority bias) (Incrementing linearly from 0.4 to 1.0), the importance sampling weight of the batch samples is calculated and used to correct the gradient update step size in subsequent steps and suppress priority sampling bias; The formula for calculating the importance sampling weight is as follows: The weights are used to weight the loss function of network training in subsequent steps, wherein, The importance sampling weight of the i-th sample is represented by N, where N represents the total sample capacity of the experience replay pool. This indicates the sample sampling probability or the normalized priority. The parameter representing the bias in balancing priority is This represents the largest weight value in the current batch.

[0036] In this embodiment, further, in step S4, the state feature enhancement of the training samples is performed, specifically including the following steps: The attention weights between the drone itself, neighboring drones, and obstacles are calculated using both distance-based exponential decay and collision risk coefficient-based weight decay. The attention weights are used to weight the environmental interaction information within the local perception range, and the weighted features are concatenated with the UAV's own motion state to obtain the enhanced state vector.

[0037] In this embodiment, further, in step S5, the local value network adopts an improved deep Q-network (Rainbow DQN) architecture, and the algorithm structure diagram is shown below. Figure 1 As shown, the construction of the Rainbow DQN value network, which integrates multiple strategies to improve value estimation accuracy and ensure training stability, includes the following steps: (1) Construct a Double DQN architecture: the main network (with 36-dimensional augmented features as input, 256-128 neurons in the hidden layer, and ReLU activation function) is responsible for action selection, and the target network (with the same structure as the main network) is responsible for value evaluation. The parameters are synchronized once every 1000 steps. The target network adopts a Dueling DQN structure. The state value branch V(s) outputs a single-value state value, and the dominance value branch A(s,a) outputs the dominance values ​​of six actions (forward, backward, left, right, up, down). The action value is calculated using the following formula: in, Indicates the value of an action. Represents the value of a state. This indicates the advantage value of action a when collected in state s. This represents the mean of the dominance values ​​for all actions. NoisyNet is constructed by adding Gaussian noise (mean 0, variance 0.1) to the fully connected layer to replace ε-greedy exploration; (2) Divide the initial training sample, i.e., the ORCA trajectory, into 4-step segments and calculate the initial distributed TD error; (3) Store the fragments into the experience replay pool according to their priority, and set the new samples as the highest priority; (4) Sample 32 batches each time, determine the optimal action for the next state through multi-step TD and Double DQN, and estimate the value distribution using the C51 algorithm (51 value bins, ranging from -10 to 10); (5) Calculate the cross-entropy loss between the main network output and the target distribution, update the parameters using the SGD algorithm (learning rate 0.001, momentum 0.9), and reset the NoisyNet noise at each step.

[0038] In this embodiment, a further schematic diagram of the Rainbow DQN algorithm combined with the QMIX network is shown below. Figure 2 As shown, step S6 specifically includes; Construct a QMIX hybrid value network, which includes a hybrid layer for inputting the local action values ​​of each UAV and a super network for inputting the global state vector; Based on the global state vector through the hypernetwork Generate non-negative weights and bias terms ; The hybrid layer is used to fuse the local motion values ​​[Q1,Q2,...,Qn] of all UAVs into a global value. The calculation formula is as follows: in, Indicates overall value. Let s represent the local value of the i-th drone, and sglobal represent the global state.

[0039] Construct a global target value network with the same structure as the QMIX hybrid value network, synchronizing its parameters with the main hybrid network every 2000 steps, and calculate the target global value using the following formula: in, express, Indicates global reward. Indicates the discount factor. Indicates the step size in multi-step learning. This indicates that the global target value network is determined based on the next state. and optimal action combination Calculated evaluation value.

[0040] In this embodiment, step S7 further includes: Construct a global loss function, and weight the prediction error of the global value according to the modified weights; By minimizing the global loss function, the parameters of both the local value network and the hybrid value network are updated simultaneously. The global loss function is expressed as: in, This indicates the number of samples in the training batch. Indicates the sample index. The importance sampling weight of the i-th sample is represented. This represents the mean square error function. Represents the global target value of the i-th sample. , Represents the global value of the i-th sample. .

[0041] This invention introduces a QMIX hybrid value network through an improved Rainbow DQN algorithm, overcoming the limitation of traditional Rainbow DQN algorithms that can only be used for single-agent training. It achieves a combination of centralized training and distributed execution, thereby optimizing cooperation among multiple UAVs. During the training phase, each UAV shares environmental information and jointly learns cooperative strategies, ensuring that the UAV swarm can make independent decisions based on its local perception state during the execution phase, while guaranteeing high efficiency in task coordination. During the execution phase, each UAV can adjust its flight path according to real-time status, avoiding single points of failure that may occur in traditional centralized control methods and enhancing the system's robustness.

[0042] Example 2 In conjunction with the method provided in Embodiment 1, this embodiment provides a multi-UAV path planning and obstacle avoidance system 21 based on the Rainbow DQN algorithm, the system structure diagram of which is shown below. Figure 3 As shown, it includes: The environment modeling module 22 is used to construct a system model containing information about multiple UAVs and the environment, and to define the local state vector and global state vector of the UAVs. The trajectory generation and sample initialization module 23 is used to generate collision-free operation trajectories of multiple UAVs, calculate the target value of trajectory segments using multi-step temporal difference formulas, and store training samples containing local state vectors, global state vectors and target values ​​into the training dataset. The priority sampling and feature enhancement module 24 is used to sample the training samples from the training dataset and perform sample importance weighting and state feature enhancement on the training samples respectively to obtain the corrected weights and enhanced state vectors. The local value network construction module 25 is used to construct a local value network for each UAV and output local action value using the enhanced state vector; The global value hybrid training module 26 is used to construct a hybrid value network, receive the local action values ​​of all UAVs, and fuse them with the global state vector to form a global value; calculate the global loss function using the corrected weights, and jointly train the local value network and the hybrid value network.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A multi-UAV path planning and obstacle avoidance method based on the Rainbow DQN algorithm, characterized in that, include: Step S1: Construct a system model that includes information on multiple drones and the environment; Step S2: Define the local state vector and global state vector of the UAV; Step S3: Generate collision-free operating trajectories for multiple UAVs, calculate the target value of trajectory segments using the multi-step temporal difference formula, and store the training samples containing local state vectors, global state vectors, and target values ​​into the training dataset. Step S4: Sample the training samples from the training dataset, and perform sample importance weighting and state feature enhancement on the training samples respectively to obtain the corrected weights and enhanced state vectors; Step S5: Construct a local value network for each UAV and output the local action value using the enhanced state vector; Step S6: Construct a hybrid value network, receive the local action values ​​of all UAVs, and fuse them with the global state vector to form a global value; Step S7: Calculate the global loss function using the corrected weights, and jointly train the local value network and the hybrid value network.

2. The multi-UAV path planning and obstacle avoidance method based on the Rainbow DQN algorithm according to claim 1, characterized in that, Step S1 includes the following specific steps: Establish a three-dimensional coordinate system as the environmental space, and define the spatial attributes of terrain, static obstacles, and dynamic obstacles; Define a collision avoidance zone for each drone. The collision avoidance zone is a spherical area with the drone itself as the origin and a preset collision avoidance threshold as the radius. Define the local perception range of each drone to obtain relative information between the drone and its surrounding environment.

3. The multi-UAV path planning and obstacle avoidance method based on the Rainbow DQN algorithm according to claim 1, characterized in that, Step S2 includes the following specific steps: The local state vector includes the UAV's own position, speed, flight azimuth angle, and the relative distance, relative speed, relative speed angle, and collision risk coefficient between the UAV and neighboring UAVs and obstacles within the local perception range. The global state vector includes the three-dimensional position coordinates of all UAVs in the environmental space and the distance information of each UAV from its target point; The formula for calculating the collision risk coefficient is as follows: in, Indicates the risk coefficient. Indicates relative distance. This indicates the angle between relative velocities.

4. The multi-UAV path planning and obstacle avoidance method based on the Rainbow DQN algorithm according to claim 1, characterized in that, Step S3 specifically includes the following steps: Define the number of drones and the target point in the environmental space, call the reciprocal collision avoidance algorithm to generate collision-free running trajectories of multiple drones, and use the running trajectories as initial training samples; Define the step size n for multi-step temporal difference learning, and decompose the initial training sample into continuous trajectory segments containing n time steps; Construct an experience replay pool to store the continuous trajectory segments, and parse each trajectory segment into a state vector sequence within n time steps; The target value of the trajectory segment is calculated using a multi-step temporal difference formula, which is expressed as follows: in, This represents the target value at time step t. to Indicates from time step t to Instant rewards Indicates the discount factor. Indicates the target network's time steps state Value assessment; The local state vector, global state vector, and target value corresponding to the trajectory segment are used to form training samples and stored in the training dataset.

5. A multi-UAV path planning and obstacle avoidance method based on the Rainbow DQN algorithm according to claim 1, characterized in that, In step S4, the training samples are weighted according to their importance. The specific steps include: Construct an experience replay pool that supports priority, storing training samples and their corresponding temporal difference errors and priorities, wherein the temporal difference error of the initial training samples is set to a preset maximum value; The sampling priority of the samples is calculated using the time-series difference error, and batch samples are extracted according to the sampling priority; The formula for calculating the sampling priority is as follows: Where P(i) represents the sampling probability, δi represents the temporal difference error of the i-th training sample, and α represents the priority weight coefficient; Calculate the importance sampling weights of the batch samples, which will be used to correct the gradient update step size in subsequent steps; The formula for calculating the importance sampling weight is as follows: in, The importance sampling weight of the i-th sample is represented by N, where N represents the total sample capacity of the experience replay pool. This indicates the sample sampling probability or the normalized priority. The parameter representing the bias in balancing priority is This represents the largest weight value in the current batch.

6. The multi-UAV path planning and obstacle avoidance method based on the Rainbow DQN algorithm according to claim 1, characterized in that, In step S4, state feature enhancement is performed on the training samples, specifically including the following steps: The attention weights between the drone itself, neighboring drones, and obstacles are calculated using both distance-based exponential decay and collision risk coefficient-based weight decay. The attention weights are used to weight the environmental interaction information within the local perception range, and the weighted features are concatenated with the UAV's own motion state to obtain the enhanced state vector.

7. A multi-UAV path planning and obstacle avoidance method based on Rainbow DQN algorithm according to claim 1, characterized in that, In step S5, the local value network adopts an improved deep Q-network architecture, which includes a main network and a target network with the same structure. Both the main network and the target network are configured with the following structure: A competitive network is used to split the output of the fully connected layer of the network into a state value branch V(s) and an advantage value branch A(s,a), and calculates the action value using the following formula: in, Indicates the value of an action. Represents the value of a state. This indicates the advantage value of action a when collected in state s. This represents the mean of the dominance values ​​for all actions. Noisy networks are used to introduce Gaussian noise parameters into fully connected layers, and the randomness of the network parameters enables an exploration strategy to replace the greedy strategy. Distributed value estimation is used to discretize the output of the action value into a probability distribution containing several value atoms; During training, the main network is used to select the optimal action in the current state, and the target network is used to evaluate the value distribution of the optimal action.

8. A multi-UAV path planning and obstacle avoidance method based on Rainbow DQN algorithm according to claim 1, characterized in that, Step S6 specifically includes: Construct a QMIX hybrid value network, which includes a hybrid layer for inputting the local action values ​​of each UAV and a super network for inputting the global state vector; Based on the global state vector through the hypernetwork Generate non-negative weights and bias terms ; The hybrid layer is used to fuse the local motion values ​​[Q1,Q2,...,Qn] of all UAVs into a global value, calculated as follows: in, Indicates overall value. Let s represent the local value of the i-th drone, and sglobal represent the global state; Construct a global target value network with the same structure as the QMIX hybrid value network to calculate the global target value. The calculation formula is as follows: in, express, Indicates global reward. Indicates the discount factor. Indicates the step size in multi-step learning. This indicates that the global target value network is determined based on the next state. and optimal action combination Calculated evaluation value.

9. A multi-UAV path planning and obstacle avoidance method based on Rainbow DQN algorithm according to claim 1, characterized in that, Step S7 specifically includes: Construct a global loss function, and weight the prediction error of the global value according to the modified weights; By minimizing the global loss function, the parameters of both the local value network and the hybrid value network are updated simultaneously. The global loss function is expressed as: in, This indicates the number of samples in the training batch. Indicates the sample index. The importance sampling weight of the i-th sample is represented. This represents the mean square error function. Represents the global target value of the i-th sample. , Represents the global value of the i-th sample. .

10. A multi-UAV path planning and obstacle avoidance system based on the Rainbow DQN algorithm, characterized in that, include: The environment modeling module is used to build a system model that includes information about multiple drones and the environment, and to define the local state vector and global state vector of the drones. The trajectory generation and sample initialization module is used to generate collision-free operating trajectories for multiple UAVs, calculate the target value of trajectory segments using multi-step temporal difference formulas, and store training samples containing local state vectors, global state vectors and target values ​​into the training dataset. The priority sampling and feature enhancement module is used to sample the training samples from the training dataset, and perform sample importance weighting and state feature enhancement on the training samples respectively to obtain the corrected weights and enhanced state vectors. The local value network construction module is used to construct a local value network for each UAV and output the local action value using the enhanced state vector; The global value hybrid training module is used to construct a hybrid value network, receive the local action values ​​of all UAVs, and fuse them with the global state vector to form a global value; it uses the corrected weights to calculate the global loss function and jointly trains the local value network and the hybrid value network.

Images