Collision avoidance method for dense flight of unmanned aerial vehicle, electronic equipment and unmanned aerial vehicle

Abstract

The invention discloses an unmanned aerial vehicle dense flight collision avoidance method, electronic equipment and an unmanned aerial vehicle. The method comprises the following steps: acquiring own state information and state information of other unmanned aerial vehicles; detecting potential collision risks with other unmanned aerial vehicles according to the obtained state information, and integrating the states of the unmanned aerial vehicles which may collide with each other; the states of other unmanned aerial vehicles in continuous time steps and the state of the unmanned aerial vehicles at the current moment are spliced to serve as input observation; extracting dynamic features according to input observation, inputting the extracted dynamic features into a pre-trained action prediction model, and outputting an angular velocity and an action decision value; determining an action executed at the current moment according to the action decision value; if the conflict risk does not exist, returning to a preset route by adopting an L1 macro action; if the conflict risk exists, the course is adjusted according to the action decision. The trajectory tracking macro action is contained in the action space, the unmanned aerial vehicle can return to the preset flight path after collision avoidance is completed, and the method can be widely applied to the technical field of airplanes.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a method for avoiding collisions of unmanned aerial vehicles during dense flight, electronic equipment and unmanned aerial vehicles. Background Art

[0002] Fixed-wing UAVs have broad application prospects in many fields due to their long flight time and high speed. With the widespread application of fixed-wing UAVs in the future, there may be a large number of UAVs flying simultaneously in local airspace. Therefore, one of the urgent problems to be solved is how to ensure the safe operation of UAVs. This requires each UAV to avoid collisions with its neighboring UAVs and minimize deviations from its preset flight path.

[0003] Since the flight paths of drones are constantly changing according to mission requirements and flight conditions, online collision avoidance methods play an important role in ensuring the flight safety of drones. In these methods, each drone makes independent decisions based on its own observations and those of neighboring drones. Therefore, these methods have the advantages of flexibility, high computational efficiency, and scalability. Traditional online collision avoidance methods can ensure a safe distance between drones in many scenarios. However, these methods have certain limitations when dealing with the collision avoidance problem when a large number of drones fly in crowded airspace. In recent years, methods based on deep reinforcement learning (DRL) have attracted widespread attention. The basic concept of DRL is that an agent learns the optimal strategy by interacting with the environment and receiving reward feedback. By using deep neural networks, agents can learn comprehensive strategies in simulated scenarios to cope with complex and dangerous situations in real environments.

[0004] At present, many studies are devoted to real-time collision avoidance methods based on DRL algorithms. However, these methods mainly focus on rotary-wing UAVs, while dynamically constrained fixed-wing UAVs are rarely considered in existing studies. Compared with rotary-wing UAVs, fixed-wing UAVs have higher flight speeds and are limited by the minimum flight speed and minimum turning radius. Therefore, the collision avoidance problem of fixed-wing UAVs is more complicated. In addition, existing studies have insufficient discussion on the characteristics of the collision avoidance problem. Most existing DRL-based methods directly incorporate the observation information of neighboring UAVs into the neural network, ignoring the relative states between UAVs that may cause collisions. Most of these methods focus on generating collision-free solutions rather than improving air traffic efficiency or completing the UAV's mission. In order to avoid collisions, UAVs may deviate significantly from the established route, and it is difficult to learn a strategy to return to the established route. Summary of the invention

[0005] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a cooperative collision avoidance method for intensive flight of drones, an electronic device, a medium and a drone based on deep reinforcement learning for direction adjustment.

[0006] The first technical solution adopted by the present invention is:

[0007] A method for avoiding collisions in densely-flying unmanned aerial vehicles, applied to unmanned aerial vehicles, comprises the following steps:

[0008] Obtaining its own status information and the status information of other drones; the status information includes position information and speed information;

[0009] Detect potential collision risks with other drones based on the acquired status information and integrate the status of drones that may collide with each other;

[0010] The states of other drones in consecutive time steps are concatenated with the current state of the drone itself as input observations;

[0011] Extract dynamic features based on input observations, input the extracted dynamic features into a pre-trained motion prediction model, and output angular velocity and motion decision values;

[0012] According to the action decision value, the action to be performed at the current moment is determined: if there is no risk of conflict, the L1 macro action is used to return to the predetermined route; if there is a risk of conflict, the heading is adjusted according to the angular velocity output by the neural network to avoid collision.

[0013] Furthermore, the detecting of potential collision risks with other drones based on the acquired status information and integrating the status of drones that may collide with each other includes:

[0014] Assume that the drone itself A i The position is p i , the speed is v i ; Other UAVs A j The position is p j , the speed is v j ; then A j Relative to A i The position and velocity are expressed as:

[0015] p ji =p j -p i

[0016] v ji =v j -v i

[0017] Define v ji The extension line is l ji; if l ji With drone A i If the safety zones of the two drones intersect, it means that the two drones will collide with each other within a certain period of time in the future;

[0018] When A i and A j When there is a collision risk, the expected collision time is defined as

[0019]

[0020] In the formula, R i A i The radius of the circular safety zone, d ji A i To extension line ji Straight-line distance;

[0021] The smaller the value, the higher the value. i With A i The higher the risk of collision; Sort the values ​​by size and obtain a certain number of neighboring drones below the threshold τ t If the number is insufficient, it is filled with a fixed format to indicate that the rest of the observation space does not contain information.

[0022] Furthermore, the state of other drones in consecutive time steps is concatenated with the state of the drone itself at the current moment as input observation, including:

[0023] At time t, UAV A i The observation of is composed of two parts, denoted as Indicates A i one's own state; Indicates the state of neighboring drones, where represents the partial observation state of the jth neighboring UAV;

[0024] Will Combined with observations from n consecutive time steps as input observations

[0025] Further, the action prediction model includes a policy network and two action value networks;

[0026] The extracting of dynamic features according to the input observation, inputting the extracted dynamic features into a pre-trained motion prediction model, and outputting angular velocity and motion decision value include:

[0027] Use convolutional neural network to extract dynamic features from input observations and obtain vector h;

[0028] The vector h is input into the policy network, which generates two parameters: and By Sampling to get the output action a t , a t Including angular velocity and action decision value;

[0029] The vector h and the action a t Estimated value of Input action value network, output action a t The value assessment of is denoted by Q.

[0030] Furthermore, the L1 macro action is calculated by an L1 navigation algorithm, specifically:

[0031]

[0032] In the formula, v i is the speed of the drone, a s Represents speed v i The vertical lateral acceleration, L1 is the distance between the drone position and the reference point on the predetermined route, and η is the vector v i The angle between L1 and ω m is the calculated angular velocity as the L1 macro action.

[0033] Furthermore, the training process of the action prediction model includes:

[0034] In each iteration, the output of the action-value network that outputs the smaller Q value is selected;

[0035] During the training cycle, the drone follows the policy π θ Run T time steps in the environment, store the observation, reward and action data of each time step t in the buffer R, and use it to update the parameters of the policy network and action value network;

[0036] The action prediction model adds an entropy term to its objective function, expressed as:

[0037]

[0038] In the formula, H(π θ (·|o t )) represents the strategy π θ In the observation t The degree of randomness under; α is the regularization coefficient; r(o t ,a t ) means in state o t Select action a t The reward value obtained; According to the strategy π θ The expected value computed from the probability distribution of state-action pairs.

[0039] Furthermore, the reward function during training is as follows:

[0040] r=r w +r c

[0041]

[0042] In the formula, r w is the runtime reward, r c is the collision penalty; k ε , k η and k c These are all hyperparameters; is the tracking error at time t; η t is the angle between the speed direction of the UAV and the direction of L1 at time t; p i A i The position at time t; p j A j The position at time t; R i A i The radius of the circular safe area.

[0043] Furthermore, the calculation formula for the value loss during training is as follows:

[0044]

[0045] The formula for calculating the strategy loss is as follows:

[0046]

[0047] In the formula, B is the batch size of data collected from the experience pool, y t is the target Q value calculated based on the Bellman equation, Q ωj The value evaluation output by the action value network.

[0048] The second technical solution adopted by the present invention is:

[0049] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, the code set or the instruction set is loaded and executed by the processor to implement a method for avoiding collisions during dense flight of unmanned aerial vehicles as described above.

[0050] The third technical solution adopted by the present invention is:

[0051] A drone comprises the electronic device as described above.

[0052] The fourth technical solution adopted by the present invention is:

[0053] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a method for avoiding collisions during dense flight of unmanned aerial vehicles as described above.

[0054] The fifth technical solution adopted by the present invention is:

[0055] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned method for avoiding collisions in densely packed flight of unmanned aerial vehicles.

[0056] The beneficial effect of the present invention is that the present invention incorporates the trajectory tracking macro action into the action space of the proposed collision avoidance algorithm based on deep reinforcement learning. After completing the collision avoidance, the UAV can return to the preset flight path according to the L1 macro action, which can effectively ensure the safety and orderly flight of a large number of UAVs when flying at close range. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 It is a schematic diagram of the collision risk model of two fixed-wing UAVs;

[0059] Figure 2 is a schematic diagram of the system functional architecture in an embodiment of the present invention;

[0060] Figure 3 is a schematic diagram of the relative relationship between the drones in an embodiment of the present invention;

[0061] Figure 4 is a schematic diagram of an L1 navigation algorithm in an embodiment of the present invention;

[0062] Figure 5 is a schematic diagram of a neural network architecture in an embodiment of the present invention;

[0063] Figure 6 is a schematic diagram of a strategy training algorithm in an embodiment of the present invention;

[0064] Figure 7 is an overall flow chart of the UAV intensive flight collision avoidance algorithm in an embodiment of the present invention;

[0065] Figure 8 The present invention is a flowchart of the steps of a method for avoiding collisions during intensive flight of unmanned aerial vehicles in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0067] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0068] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0069] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention in combination with the specific content of the technical solution.

[0070] In view of the fact that the centralized optimization coordination method is difficult to be effectively applied when there are a large number of drones in the local airspace, the present invention proposes a decentralized collision avoidance method. Considering the characteristics of the collision avoidance problem, the present invention proposes a collision prediction method to identify adjacent drones with significant threats, and designs a convolutional neural network to extract dynamic environmental features in continuous time steps. Secondly, the trajectory tracking macro action is incorporated into the action space of the proposed collision avoidance algorithm based on deep reinforcement learning. Under the guidance of the reward function that considers the reward for approaching the preset flight path, the drone can return to the preset flight path according to the L1 macro action after completing the collision avoidance. The proposed method is trained in a simulation scenario, and the SoftActor-Critic (SAC) algorithm is used to implement the model weight parameter update. Such a coordination strategy can ensure the safe and orderly flight of a large number of drones when flying at close range.

[0071] Example 1

[0072] like Figure 7 and Figure 8 As shown, this embodiment provides a method for avoiding collisions in densely packed flight of unmanned aerial vehicles, which is applied to unmanned aerial vehicles and includes the following steps:

[0073] S1. Obtaining the status information of the drone and other drones; the status information includes position information and speed information;

[0074] S2, detecting potential collision risks with other UAVs based on the acquired status information, and integrating the status of UAVs that may collide with each other;

[0075] S3, concatenate the states of other drones in consecutive time steps with their own current state as input observations;

[0076] S4, extracting dynamic features according to the input observations, inputting the extracted dynamic features into a pre-trained motion prediction model, and outputting angular velocity and motion decision values;

[0077] S5. Determine the action to be performed at the current moment according to the action decision value: if there is no risk of conflict, use the L1 macro action to return to the predetermined route; if there is a risk of conflict, adjust the heading according to the angular velocity output by the neural network to avoid collision.

[0078] The working principle of the method of this embodiment is as follows: During the flight along their respective routes, the drones first sense the relative positions and speeds of other drones in the surrounding airspace. Then, a circle with its own position as the center and a radius of R is set for each drone. i The circular safety area is constructed and the potential collision risk with other drones is detected according to the extension line law. The risk between the drones that may collide is evaluated by estimating the expected collision time.

[0079] Next, the states of the drones that may collide are integrated according to the expected collision time. The states of other drones in consecutive time steps are concatenated with the current state of the drone itself, and high-dimensional features (i.e. dynamic features) are extracted through two convolutional layers. These high-dimensional features are then passed to the pre-trained deep neural network (i.e. action prediction model) for decision-making and output of the corresponding action.

[0080] Finally, based on the action decision value given by the neural network, the drone determines the action that should be performed at the current moment. If there is no risk of conflict, the drone will use the L1 macro action to return to the predetermined route; if there is a risk of conflict, the drone will adjust its heading according to the angular velocity output by the neural network to avoid collision.

[0081] The above method is explained in detail below with reference to the accompanying drawings and specific implementation methods.

[0082] (1) Kinematic model of fixed-wing UAV

[0083] Assume that there are N fixed-wing drones in the local airspace, and each drone A i , the model of i∈{1,...,N} is simplified to a non-holonomic motion model. i , i∈N, its dynamic characteristics can be described by the Cartesian coordinate system. This embodiment mainly studies the conflict resolution problem of UAVs in two-dimensional space. Formula (1) defines A i Motion state in two-dimensional space:

[0084]

[0085] where p i =[x i ,y i ] T and v i Indicates A i Position and velocity in the inertial coordinate system. φ i Represents the yaw angle. ω i is the yaw angular velocity. i Subject to the maximum yaw rate ω max constraints.

[0086] To ensure collision-free flight of UAVs, the collision risk is predicted based on the motion states of adjacent UAVs. i A monopolized circular safety zone is defined, which is p i The center is R, and the radius is i , represented by its shape and dynamics as D i (p i ,R i ). If A j Enter Di (p i ,R i ), then A i and A j In collision state.

[0087] (2) Geometric analysis of conflict resolution

[0088] According to A i and A j The motion state of the object is used to predict the collision risk. Figure 1 As shown, A j Relative to A i The position and velocity can be expressed as:

[0089]

[0090] Define v ji The extension line is l ji ; if l ji With D i (p i ,R i ) intersect, it means that the two drones will collide with each other in the future. It is defined as formula (3):

[0091]

[0092] when When A i and A j There is a risk of collision, define the expected collision time as follows:

[0093]

[0094] here Contains about v ji Information and information about p ji information, which indicates that A i and A j Time to collide with each other. Figure 1 If A j Entering the shadow area, the two drones will collide. j Arrival P cl The length of time indicates the risk of collision.

[0095] The collision avoidance problem of fixed-wing UAVs can be solved as an optimization problem, whose goal is to minimize the deviation of each UAV from its preset flight path under the constraint of safe separation. Here, this embodiment uses a series of waypoints to represent the preset flight path of each UAV.i Observation information can be obtained And use the trained strategy network to calculate the collision avoidance control instructions

[0096] At time t, A i The observation of is composed of two parts, denoted as in, Indicates A i The state of the drone itself can be fully observed. The other part is the state of the neighboring drones, which can only be partially observed and is defined as in Represents the partial observation state of the jth neighboring UAV.

[0097] Since the neighboring drones are in different states, i The impact of is also different, so A j The focus should be on drones that pose a greater threat to them. The smaller the collision risk, the higher the collision risk. We select m neighboring drones. The value is the smallest and below the threshold τ t (where τ t The observation status of some UAVs (for the forward time interval) j=1…m, as A i input observations; if the number of neighboring drones that meet the requirements is less than m, the input observations are filled in a fixed format, indicating that the rest of the observation space does not contain information. This approach of focusing on high-risk collision hazards helps filter out the impact of collision risks of some non-urgent drones, thereby achieving safe and orderly flight from local to global. At the same time, this also reduces the computational cost of the collision avoidance method.

[0098] Will Combined with observations from n consecutive time steps, the input observations are defined Given an input observation Policy neural network π parameterized by θ θ Output action According to the previous discussion, the output is UAV A i The angular velocity and action decision value of each drone should minimize the deviation from its preset flight path. The optimal strategy can be obtained by solving the following constrained optimization problem:

[0099]

[0100] in, represents the tracking error, which represents the θ Controlled drone A iThe distance between the actual position and the target path at time t.

[0101] (3) Optimized design

[0102] This embodiment regards the multi-UAV collision avoidance problem as a partially observable sequential decision-making problem and applies the deep reinforcement learning (DRL) algorithm to solve this problem. In this section, the DRL-based collision avoidance method architecture will be described in detail. Figure 2 The overall architecture of the improved method is presented in Figure 2 In the proposed algorithm, each UAV obtains the status of other UAVs to predict the risk of collision; the deep reinforcement learning (DRL) network uses the observation data of high-risk UAVs and reference points on the preset flight path to decide the action. This action may be a basic action calculated by the neural network or a macro action calculated by the tracking controller.

[0103] 3.1) Design of key modules of deep reinforcement learning

[0104] a) Observation space: As mentioned above, observation Include and Two parts, including is the observation of the drone’s own state, and Contains the drone A i Observations of neighboring drones with high collision risk. We use a collision risk model to capture the relationship between drones. middle, It is a drone A i The speed (including the speed v i and direction ),and is the reference point relative to drone A i The polar coordinates of the position (distance L1 and angle η). The reference point is used to guide each drone to follow a preset flight path. It depends on p i (Look-ahead distance L1 and preset flight path). L1 can be determined according to formula (6):

[0105]

[0106] Among them, k l is a hyperparameter used to tune the look-ahead distance.

[0107] In observation middle, Indicates the neighboring drone A j observation information. It is a drone A j The speed (including the speed v j and direction Φj ),and It is a drone A j Relative to UAV A i The polar coordinates (including the distance d ji and angle α ji ). Detailed information of the observation is as follows Figure 3 And formula (7) shows:

[0108]

[0109] exist Figure 3 In, x g Represents the x-axis of the ground coordinate system. The reference point is a point on the preset flight path that is close to the UAV A. i The distance between them is L1. L1 is determined by ε i and v i Joint decision.

[0110] When with drone A i When the number of drones at risk of collision is less than m, the remaining (i.e., the part of the observation space that does not record the drone state) is set to [0,0,-1,-1], indicating that the rest of the observation space contains no information.

[0111]

[0112] Each drone can observe the motion status of the surrounding drones at the current moment, and combine historical observation information with its own motion status to use the observation space They use this information to assess the impact of their actions on their surroundings, and thus learn how to avoid other drones. So, with this information, they can more efficiently determine the appropriate actions.

[0113] By combining the first n consecutive observations To construct the input observation space here, Represents the observation data of drone i at time point t. By combining the observation data of multiple consecutive time points, we can provide a more informative input for the neural network, which helps the network understand the changing trend of the dynamic environment and make more accurate decisions accordingly.

[0114] b) Action space: The action space is carefully designed so that the agent learns to return to the preset flight path and avoid collisions. It consists of two parts: basic actions and action decision values. The basic action is the yaw angular velocity ω b , which is directly generated by the deep reinforcement learning (DRL) algorithm based on specific observations. The action decision value is used to decide the control instructions of the drone. Therefore, the action space is defined as (9):

[0115]

[0116] Among them, c is the control flag of L1 macro action, indicating that agent A i Whether L1 macro action is currently adopted. c=0 means A i Take the basic action ω directly output by the DRL algorithm b ; c=1 means A i Take the L1 macro action ω output by the path-following L1 navigation algorithm m . L1 navigation algorithm such as Figure 4 The main parameters are defined as in formula (10).

[0117]

[0118] Where η is the vector v i The angle between L1 and s Represents speed v i The vertical lateral acceleration, R is the radius of the arc.

[0119] By adding macro actions to the action space, the drone is not only able to learn how to avoid collisions, but also to autonomously return to the preset flight path. In addition, it reduces action oscillations caused by exploration randomness. This ensures smooth flight and stable flight guidance.

[0120] c) Reward Design: A reward function is designed that takes into account the safety interval requirements between drones and the minimization of A i The deviation from the preset flight path. The reward function consists of two parts: runtime reward r w and collision penalty r c .

[0121] r=r w +r c (11)

[0122] Different from the existing reward function, which mainly considers minimizing the distance between each drone position and its target point, the r w (Runtime Reward) will guide the drone to follow the preset flight path more accurately. w Defined as:

[0123]

[0124] Collision Penalty c Used to make A i Learn to avoid collisions. Therefore, r c Designed for:

[0125]

[0126] where k ε , k η and k c are all hyperparameters; η t is the angle between the speed direction of the UAV and the direction of L1 at time t; p i A i The position at time t; p j A j The position at time t; R i A i The radius of the circular safe area.

[0127] 3.2) Neural Network Design

[0128] The neural network of the deep reinforcement learning (DRL) algorithm is designed based on the actor-critic architecture. In addition, a convolutional neural network (CNN) module is used to process the observation data. The architecture of the neural network is as follows Figure 5 shown.

[0129] In each step, the observation data of n consecutive moments are spliced ​​together and convolved in a convolutional neural network (CNN) to generate a fixed-size vector h. The CNN module can mine the hidden relationship between the action and the state of neighboring drones from the continuous observation data and extract dynamic features. The vector h will be input into the policy network (Actor) and the action value network (Critic). Specifically, the hidden layer of the policy network Actor consists of two fully connected layers with 256 neurons. The policy network Actor generates two parameters: and and by Sampling to get the output action a t Then, the vector h and the action a t Estimated value of It will be input into the action value network Critic. Specifically, the action value network Critic consists of two fully connected layers with 256 neurons, which will output the action a t The value assessment of is denoted by Q.

[0130] 3.3) Neural Network Training

[0131] The Soft Actor-Critic algorithm is used to train and update the multi-UAV collision avoidance policy network and action value network. Figure 6, the detailed steps of the training algorithm are shown in Algorithm 1. The Soft Actor-Critic (SAC) algorithm is an offline policy algorithm that uses a random strategy and has the characteristics of high sample utilization and high exploration efficiency. In the SAC algorithm, there are two action value networks and a policy network, and each action value network has a corresponding target network. By soft-updating the parameters of the target network, the training process can be more stable. In each iteration, the output of the action value network that outputs a smaller Q value is selected. This can alleviate the problem of over-estimation of the Q value. During the training cycle, each drone is trained according to the policy π θ Run T time steps in the environment. Data including observations, rewards, and actions at each time step t are stored in a buffer R for policy updates. The policy adds an entropy term to its objective function to encourage exploratory behavior. The policy objective function can be defined as:

[0132]

[0133] In the formula, H(π θ (·|o t )) represents the strategy π θ In the observation t The regularization coefficient α is used to control the importance of entropy. The entropy regularization method increases the degree of exploration of the reinforcement learning algorithm.

[0134] Reparameterization techniques are used to sample actions, update policy networks and action-value networks. Through an iterative training process consisting mainly of policy evaluation and policy update, neural network weight parameters that can ensure good performance of deep reinforcement learning-based collision avoidance methods can be obtained.

[0135] The calculation formula for value loss is as follows:

[0136]

[0137] The formula for calculating the strategy loss is as follows:

[0138]

[0139] Example 2

[0140] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 8 A method for avoiding collisions during dense flight of unmanned aerial vehicles is shown.

[0141] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0142] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0143] Since the electronic device is an electronic device corresponding to a method for avoiding collisions during intensive flight of unmanned aerial vehicles in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0144] Example 3

[0145] This embodiment provides a drone, which is equipped with a device for performing the following steps: Figure 8 The electronic device of the method shown. Specifically, the drone can be a fixed-wing drone or other drones.

[0146] Since the UAV is a UAV corresponding to the UAV intensive flight collision avoidance method in an embodiment of the present invention, the principle of solving the problem is similar to that of the method. Since it has corresponding functions and beneficial effects, reference can be made to the implementation process of the above method embodiment, and the repeated parts will not be repeated.

[0147] Example 4

[0148] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 8 A method for avoiding collisions during dense flight of unmanned aerial vehicles is shown.

[0149] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0150] Since the storage medium is a storage medium corresponding to a method for avoiding collisions in intensive flight of unmanned aerial vehicles in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0151] Example 5

[0152] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a method for avoiding collisions in dense flight of unmanned aerial vehicles according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing various embodiments may be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0153] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0154] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0155] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for avoiding collisions when unmanned aerial vehicles are flying in dense concentration, characterized in that: Applied to UAV, it includes the following steps: Obtaining its own status information and the status information of other drones; the status information includes position information and speed information; Detect potential collision risks with other drones based on the acquired status information and integrate the status of drones that may collide with each other; The states of other drones in consecutive time steps are concatenated with the current state of the drone itself as input observations; Extract dynamic features based on input observations, input the extracted dynamic features into a pre-trained motion prediction model, and output angular velocity and motion decision values; According to the action decision value, the action to be performed at the current moment is determined: if there is no risk of conflict, the L1 macro action is used to return to the predetermined route; if there is a risk of conflict, the heading is adjusted according to the angular velocity output by the neural network to avoid collision.

2. The method for avoiding collisions when unmanned aerial vehicles are flying in dense concentration according to claim 1, characterized in that: The method of detecting potential collision risks with other drones based on the acquired status information and integrating the status of drones that may collide with each other includes: Assume that the drone itself A i The position is p i , the speed is v i ; Other UAVs A j The position is p j , the speed is v j ; then A j Relative to A i The position and velocity are expressed as: p ji =p j -p i v ji =v j -v i Define v ji The extension line is l ji ; if l ji With drone A i If the safety zones of the two drones intersect, it means that the two drones will collide with each other within a certain period of time in the future; When A i and A j When there is a collision risk, the expected collision time is defined as In the formula, R i A i The radius of the circular safety zone, d ji A i To extension line ji Straight-line distance; The smaller the value, the higher the value. i With A j The higher the risk of collision, the Sort the values ​​by size and obtain a certain number of neighboring drones below the threshold τ t If the number is insufficient, it is filled with a fixed format to indicate that the rest of the observation space does not contain information.

3. The method for avoiding collisions when unmanned aerial vehicles are flying in dense concentration according to claim 1, characterized in that: The state of other drones in consecutive time steps is concatenated with the state of the drone itself at the current moment as input observation, including: At time t, UAV A i The observation of is composed of two parts, denoted as Indicates A i one's own state; Indicates the state of neighboring drones, where represents the partial observation state of the jth neighboring UAV; Will Combined with observations from n consecutive time steps as input observations 4. The method for avoiding collisions of UAVs in dense flight according to claim 1, characterized in that: The action prediction model includes a policy network and two action value networks; The extracting of dynamic features according to the input observation, inputting the extracted dynamic features into a pre-trained motion prediction model, and outputting angular velocity and motion decision value include: Use convolutional neural network to extract dynamic features from input observations and obtain vector h; The vector h is input into the policy network, which generates two parameters: and By Sampling to get the output action a t , a t Including angular velocity and action decision value; The vector h and the action a t Estimated value of Input action value network, output action a t The value assessment of is denoted by Q.

5. The method for avoiding collisions of UAVs in dense flight according to claim 1, characterized in that: The L1 macro action is calculated by the L1 navigation algorithm, specifically: In the formula, v i is the speed of the drone, a s Represents speed v i The vertical lateral acceleration, L1 is the distance between the drone position and the reference point on the predetermined route, and η is the vector v i The angle between L1 and ω m is the calculated angular velocity as the L1 macro action.

6. The method for avoiding collisions of UAVs in dense flight according to claim 4, characterized in that: The training process of the action prediction model includes: In each iteration, the output of the action-value network that outputs the smaller Q value is selected; During the training cycle, the drone follows the policy π θ Run T time steps in the environment, store the observation, reward and action data of each time step t in the buffer R, and use it to update the parameters of the policy network and action value network; The action prediction model adds an entropy term to its objective function, expressed as: In the formula, H(π θ (·|o t )) represents the strategy π θ In the observation t The degree of randomness under; α is the regularization coefficient; r(o t ,a t ) means in state o t Select action a t The reward value obtained; According to the strategy π θ The expected value computed from the probability distribution of state-action pairs.

7. The method for avoiding collisions of UAVs in dense flight according to claim 6, characterized in that: The reward function during training is as follows: r＝r w +r c In the formula, r w is the runtime reward, r c is the collision penalty; k ε , k η and k c These are all hyperparameters; is the tracking error at time t; η t is the angle between the speed direction of the UAV and the direction of L1 at time t; p i A i The position at time t; p j A j The position at time t; R i A i The radius of the circular safe area.

8. The method for avoiding collisions of UAVs in dense flight according to claim 6, characterized in that: The calculation formula of value loss during training is as follows: The formula for calculating the strategy loss is as follows: In the formula, B is the batch size of data collected from the experience pool, y t is the target Q value calculated based on the Bellman equation, Q ωj The value evaluation output by the action value network.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 8.

10. A drone, characterized in that: Comprising an electronic device as claimed in claim 9.

Images