A multi-unmanned aerial vehicle local dynamic obstacle avoidance method, device and storage medium

Abstract

The application discloses a kind of multi-unmanned plane local dynamic obstacle avoidance method, device and storage medium, wherein method includes: based on Markov decision process, formalization modeling is carried out to distributed multi-unmanned plane dynamic obstacle avoidance problem;Obtain point cloud data by laser radar carried on each unmanned plane, point cloud data is processed, and obtain obstacle information;Deep neural network is constructed, to complete the mapping of observation input to action output and the method of network updating;Wherein observation input is obtained obstacle information;Based on ORCA algorithm, deep neural network is trained, and the model after training is used for multi-unmanned plane local dynamic obstacle avoidance.The application uses dynamic obstacle avoidance algorithm based on geometry method and deep reinforcement learning method as the obstacle avoidance method of multi-unmanned plane system, effectively solves multi-unmanned plane local path planning and obstacle avoidance problem.The application can be widely applied in multi-unmanned plane local dynamic obstacle avoidance and path planning field.

Description

Technical Field

[0001] This invention relates to the field of local dynamic obstacle avoidance and path planning for multiple unmanned aerial vehicles (UAVs), and particularly to a method, apparatus and storage medium for local dynamic obstacle avoidance for multiple UAVs. Background Technology

[0002] Currently, drones are increasingly favored and deployed in various fields due to their small size, light weight, and high flexibility. However, while single-drone technology has solved many problems, it still suffers from drawbacks compared to multi-drone systems, such as weaker computing power, smaller sensing range, and weaker anti-interference capabilities, which greatly limits the application scenarios of single drones. For multi-drone systems, navigation planning is a crucial foundation for completing other tasks, and obstacle avoidance is an indispensable part of this process. How to promptly perceive and respond to unexpected obstacles during multi-drone flight, ultimately achieving the goal of path planning, is a key research focus.

[0003] In terms of obstacle perception, current multi-UAV obstacle avoidance systems mainly use passive sensors such as monocular cameras to perceive obstacles in the surrounding environment. They obtain real-time information about obstacles by acquiring real-time images and extracting feature information from them. However, such methods are not very real-time and are not accurate enough in measuring obstacle avoidance sensitive information such as distance. On the other hand, active sensors such as lidar often perform better in terms of real-time performance and accuracy because they actively emit radiation to detect and can obtain the position of reflection points.

[0004] Previous research has categorized multi-UAV obstacle avoidance algorithms into two types: model-based global path planning algorithms and local observation-based dynamic obstacle avoidance algorithms. Unlike global path planning, local dynamic obstacle avoidance does not require prior knowledge of the global environment. Instead, during flight, the UAV uses onboard sensors to perceive changes in the surrounding environment and dynamically adjusts its flight trajectory and attitude. This type of algorithm is more suitable for dynamic environments. Commonly used dynamic obstacle avoidance algorithms include artificial potential field methods, intelligent optimization algorithms represented by ant colony optimization, and machine learning algorithms. Among these, artificial potential field methods are prone to getting trapped in local minima and failing to reach the target point, and they perform poorly in some dynamic environments. Intelligent optimization algorithms such as ant colony optimization have high computational and memory requirements, making it difficult to meet real-time requirements when computational load increases suddenly. The convergence speed of machine learning algorithms is a major issue, especially since some studies directly use LiDAR readings as input for end-to-end training, which makes the algorithm even more difficult to converge. Summary of the Invention

[0005] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a method, device and storage medium for local dynamic obstacle avoidance of multiple unmanned aerial vehicles based on deep reinforcement learning algorithm.

[0006] The technical solution adopted in this invention is:

[0007] A multi-UAV local dynamic obstacle avoidance method includes the following steps:

[0008] Formal modeling of distributed multi-UAV dynamic obstacle avoidance problem based on Markov decision process;

[0009] Point cloud data is acquired by using lidar mounted on each drone, and obstacle information is obtained by processing the point cloud data.

[0010] A method for constructing a deep neural network to complete the mapping from observation input to action output and network update is proposed; where the observation input is the obtained obstacle information.

[0011] The deep neural network is trained based on the ORCA algorithm, and the trained model is used for local dynamic obstacle avoidance of multiple UAVs.

[0012] Furthermore, the formal modeling of the distributed multi-UAV dynamic obstacle avoidance problem based on Markov decision processes includes:

[0013] The problem of distributed drones detecting obstacles, avoiding obstacles, and finally reaching the target point is modeled as a Markov decision process.

[0014] The Markov decision process is represented using a six-tuple, i.e. in This refers to the state space. This refers to the action space. This refers to the state transition model. This refers to the reward function, and Ω refers to the observation space (o∈Ω). It is a multi-unmanned aerial vehicle system status. The observed probability distribution.

[0015] Furthermore, for each drone in a multi-drone system, the drone observation space at time T is defined as o. t It contains the following part: the position of the i-th drone itself. (where i∈{1,2,…n}, and n is the number of drones), the target point location tar of the i-th drone. i The obtained observed environmental information was processed. (This includes obstacle positions, speeds, etc.); the drone's action space is the speed selected within the selectable speed space, where the selectable speed space is a continuous space, meaning the action design is...

[0016] Furthermore, the tasks of the multi-UAV system include the following components:

[0017] Multiple drones should reach the target point as quickly as possible, i.e., shorten the average arrival time;

[0018] Ensure navigation safety during the journey to the target point, that is, prevent collisions between drones or with obstacles;

[0019] The reward function is designed as follows:

[0020]

[0021] in, The rewards include obstacle avoidance rewards. As a reward for the goal, The time reward is specifically expressed as follows:

[0022] Obstacle Avoidance Rewards The negative reward is determined by the distance between the drone and the obstacle. Within a preset range, the closer the drone is to the obstacle, the greater the negative reward; the farther away it is, the smaller the reward. If a collision occurs, a fixed negative reward is given and the round ends.

[0023] Target Rewards A positive reward is given when the drone approaches the target point, and the round ends upon reaching the target point with a fixed reward. Conversely, when the drone moves away from the target point, a negative reward is given proportionally based on the distance between the drone and the target point.

[0024] Time Rewards The shorter the time it takes for a drone to reach the target point, the smaller the penalty; in other words, the negative reward is directly proportional to the arrival time.

[0025] Furthermore, the obstacle information includes static obstacle information and dynamic obstacle information;

[0026] Point cloud data is acquired by LiDAR mounted on each drone, and this data is processed to obtain obstacle information, including:

[0027] Clustering is performed on the obtained point cloud data to identify reflection points belonging to the same obstacle within the point cloud data;

[0028] After obtaining the clustering results, the center point and cluster radius of each cluster are obtained based on the coordinates of each point in each cluster, which are used as the position and size of the obstacle;

[0029] Static obstacle information is obtained from single-frame point cloud data, and dynamic obstacle information is obtained by combining multi-frame point cloud data.

[0030] Furthermore, the clustering process for the obtained point cloud data includes:

[0031] Euclidean clustering algorithm is used to cluster point cloud data; the clustering criterion is whether the Euclidean distance between each point in space is less than a preset threshold.

[0032] This preset threshold is determined in the following way:

[0033]

[0034] In the formula, X i Y i Let X represent the coordinates of the i-th point. lidar Y lidar The coordinates of the lidar are represented by α and β, which are coefficients.

[0035] The process of obtaining obstacle dynamic information by combining multi-frame point cloud data includes:

[0036] Calculate the correlation between multiple frames of point cloud data, perform data association based on the correlation, obtain the position and velocity changes as measurement values, input the measurement values ​​into the extended Kalman filter algorithm to reduce measurement errors and obtain obstacle dynamic information;

[0037] The correlation degree is calculated as follows:

[0038]

[0039] In the formula, ΔL, ΔD, and ΔI represent the position difference, size difference, and reflection intensity difference between two clusters in two frames, respectively; k1, k2, and k3 are coefficients.

[0040] Furthermore, the training is based on a dual-delay deep deterministic policy gradient algorithm to generate action decisions in a continuous action space.

[0041] The mapping from the observation space to the action space is accomplished by a deep neural network, whose network structure is mainly composed of fully connected layers; during the training phase, the actions generated by the decision network are sampled after adding noise to explore a larger space.

[0042] Furthermore, the training of the deep neural network based on the ORCA algorithm includes:

[0043] Input environmental information into the ORCA algorithm to obtain the output action;

[0044] The output action and reward are combined as a tuple and added to the experience pool as successful obstacle avoidance experience, so that the training process can quickly converge to a positive reward.

[0045] Another technical solution adopted in this invention is:

[0046] A multi-UAV local dynamic obstacle avoidance device, comprising:

[0047] At least one processor;

[0048] At least one memory for storing at least one program;

[0049] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0050] Another technical solution adopted in this invention is:

[0051] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, performs the method described above.

[0052] The beneficial effects of this invention are: This invention processes the data read by the lidar, obtains the obstacle information, and then uses it as the input of the obstacle avoidance algorithm. At the same time, it uses a dynamic obstacle avoidance algorithm based on a combination of geometric methods and deep reinforcement learning methods as the obstacle avoidance method for multi-UAV systems, which accelerates the algorithm training speed and solves the local path planning and obstacle avoidance problems of multi-UAV systems. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating the local dynamic obstacle avoidance method for multiple unmanned aerial vehicles based on deep reinforcement learning in an embodiment of the present invention.

[0055] Figure 2 This is a flowchart illustrating the improved Euclidean clustering algorithm in the obstacle detection module of this invention.

[0056] Figure 3 This is a schematic diagram illustrating the principle of the decision network architecture in this embodiment of the invention;

[0057] Figure 4This is a schematic diagram of an experience-based accelerated training method based on ORCA in a specific implementation process according to an embodiment of the present invention;

[0058] Figure 5 This is a flowchart illustrating the steps of a multi-UAV local dynamic obstacle avoidance method in an embodiment of the present invention. Detailed Implementation

[0059] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0060] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0061] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0062] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0063] like Figure 1 and Figure 5 As shown, this embodiment provides a local dynamic obstacle avoidance method for multiple UAVs based on deep reinforcement learning. This method processes the data obtained from sensors and inputs it into the algorithm, while using the geometry-based ORCA algorithm as experience to accelerate training. Specifically, it includes the following steps:

[0064] S1. Formal modeling of distributed multi-UAV dynamic obstacle avoidance problem based on Markov decision process.

[0065] As an optional implementation, the formal modeling process for the entire process in step S1 is as follows:

[0066] The local path planning problem involving multiple UAVs detecting obstacles and performing dynamic obstacle avoidance can be formalized as a partially observable Markov Decision Process (POMDP). The partially observable Markov Decision Process can be represented by a six-tuple, i.e. in This refers to the state space. This refers to the action space. This refers to the state transition model. This refers to the reward function, and Ω refers to the observation space (o∈Ω). The system state is described. The observed probability distribution.

[0067] In this embodiment, we use lidar as a primary means for multiple UAVs to perceive surrounding environmental information. Simultaneously, in addition to the data acquired by the lidar, each UAV's observation space also requires observation values ​​sent from ground stations, etc. Therefore, the UAV observation space at time T is defined as o. t It contains the following part: the position of the i-th drone itself. (where i∈{1,2,…n}, and n is the number of drones), the target point location tar of the i-th drone. i The obtained observed environmental information was processed. (This includes obstacle location, speed, etc.)

[0068] The action space of a drone is the set of velocities that can be selected within a defined continuous space; the action design is as follows: Furthermore, there are upper and lower limits to the speed.

[0069] As a further improvement to the present invention, the main tasks of the multi-UAV system include the following parts:

[0070] 1) Multiple drones should reach the target point as quickly as possible, i.e., shorten the average arrival time;

[0071] 2) Ensure navigation safety during this process, i.e., prevent collisions between drones or between drones and obstacles. In this embodiment, other drones are treated as dynamic obstacles.

[0072] Therefore, in order for the multi-UAV system to successfully complete the above tasks, the deep reinforcement learning reward function used as the obstacle avoidance algorithm in this invention is designed as follows:

[0073]

[0074] The rewards received by each drone include obstacle avoidance rewards. Target Rewards and time rewards Specifically, it is expressed as follows:

[0075] a) Obstacle Avoidance Rewards The negative reward is determined by the distance between the drone and the obstacle. Within a certain range, the closer the drone is to the obstacle, the greater the negative reward; the farther away it is, the smaller the negative reward. If a collision occurs, a fixed negative reward is given and the round ends.

[0076]

[0077] b) Target Rewards A positive reward is given when the drone approaches the target point, and the round ends upon reaching the target point with a fixed reward.

[0078]

[0079] c) Time Rewards The shorter the time it takes for the drone to reach the target point, the smaller the penalty; in other words, the negative reward is directly proportional to the arrival time.

[0080]

[0081] S2. Obtain point cloud data by using the lidar mounted on each drone, process the point cloud data, and obtain obstacle information.

[0082] Each UAV needs to process the point cloud data obtained from its onboard LiDAR to obtain a vectorized representation of the geometry and motion information of static and dynamic obstacles. This vectorized representation serves as a way to represent the multi-UAV state information and also as the state input for the obstacle avoidance algorithm. As an optional implementation, step S2 specifically includes steps S21-S23:

[0083] S21. An improved Euclidean clustering algorithm is used to cluster the point cloud data. The purpose of this step is to obtain the reflection points belonging to the same obstacle from the messy point cloud data. The specific clustering basis is whether the Euclidean distance of each point in space is less than a threshold value, which is a specific value determined in advance.

[0084] Traditional Euclidean clustering algorithms place high demands on the selection of distance thresholds. If the distance threshold is not chosen correctly, errors in detecting nearby targets or omissions in detecting distant targets often occur when processing point cloud data with uneven density. Based on the above analysis, it can be seen that the factors affecting the performance of Euclidean clustering algorithms are related to the relationship between the distance between the point cloud coordinates and the UAV coordinates and the threshold. Therefore, to address this issue, we propose a method for determining the threshold based on the distance between the point cloud and the UAV, as follows:

[0085]

[0086] Where X i Y i Let X represent the coordinates of the i-th point. lidar Y lidar The coordinates of the lidar are represented by α and β, which are related to the lidar's angular resolution and were determined through multiple experiments. The overall flow of the improved algorithm is as follows: Figure 2 As shown.

[0087] S22. After obtaining the above clusters, the center point and cluster radius of each cluster are obtained based on the coordinates of each point in each cluster, which are used as the position and size of the obstacle.

[0088] S23. After obtaining the static information of the obstacle from a single frame of point cloud data, it is also necessary to correlate multiple frames of point cloud data to obtain the dynamic information of the obstacle. The main factors for analyzing this correlation in this invention include the three-dimensional parameters of the target (radius of the enclosing sphere, position of the center point) and the laser intensity. The calculation method is as follows:

[0089]

[0090] Where ΔL, ΔD, and Δ represent the position difference, size difference (i.e., radius), and reflection intensity difference between two clusters in two frames, respectively. The correlation is inversely proportional to these differences; the larger the difference, the smaller the correlation. After data association, the obtained position and velocity changes are used as measurements and input into the extended Kalman filter algorithm to reduce measurement errors and obtain more accurate obstacle dynamic information.

[0091] S3. Construct a deep neural network to complete the mapping from observation input to action output and the method for network update; where the observation input is the obtained obstacle information.

[0092] A method for constructing a deep neural network to complete the mapping from observed input to action output and network updates, wherein the observed input is the information obtained in step S2. As an optional implementation, in step S3, the deep reinforcement learning algorithm includes the following steps:

[0093] This embodiment is based on the Twin Delayed Deep Deterministicpolicy gradient algorithm (TD3), which is a deterministic policy reinforcement learning algorithm suitable for high-dimensional continuous action spaces.

[0094] During the training phase, the TD3 algorithm explores a larger space by sampling the actions generated by the decision network after adding noise. The main algorithm flow is as follows:

[0095] S31. Initialize the parameters of the critic network and actor network.

[0096] S32. Repeat the following steps:

[0097] a) Selecting an action from a strategy t Interact with the environment and store experiences {o t ,a t ,r t+1 ,o t+1 Add it to the experience pool;

[0098] b) Sample a batch of experience {o} from the experience pool t ,a t ,r t+1 ,o t+1 Updated:

[0099] c) Update critic network parameters Update actor network parameters according to the gradient of the deterministic policy:

[0100] In a specific implementation, in step S3, the mapping from the observation space to the action space is completed by a deep neural network, wherein the network structure mainly consists of fully connected layers, and the overall network structure is as follows: Figure 3 As shown.

[0101] S4. Train the deep neural network based on the ORCA algorithm, and use the trained model for local dynamic obstacle avoidance of multiple UAVs.

[0102] During the algorithm training phase, actions obtained from the geometrically based ORCA algorithm are combined with observation and reward values ​​as experience to accelerate training. Simultaneously, later exploration improves the training paths based on early experience. The algorithm is advanced using a combination of centralized training and distributed execution, employing multi-machine obstacle avoidance actions generated based on ORCA as experience to accelerate algorithm convergence.

[0103] As an optional implementation, in step S4, the multi-UAV dynamic obstacle avoidance algorithm incorporating ORCA (Optimal Reciprocal Collision Avoidance) includes: Sampling efficiency has always been one of the unavoidable drawbacks of deep reinforcement learning. This problem can prolong the algorithm's training time and even prevent the training of a good control strategy. To address this issue, this embodiment of the invention proposes a training acceleration method based on ORCA. In this method, the experience gained from ORCA is added to the experience pool in the early stage of training to accelerate the algorithm's acquisition of positive rewards. In the later stage, further exploration is conducted based on this experience to obtain a better path.

[0104] The ORCA algorithm is a geometry-based algorithm designed from the outset for multi-agent obstacle avoidance. It is a decentralized reactive obstacle avoidance algorithm, meaning it can plan a trajectory using only a series of geometric constraints constructed based on sensors and convex optimization. However, the problem with this algorithm is that the obtained path can only guarantee no collisions, and the path and velocity selected by the algorithm are relatively conservative, leaving room for optimization in the final selected path. Therefore, we rely on this algorithm to first obtain a collision-free path, and then explore further optimization of this path.

[0105] In this embodiment of the invention, the algorithm is advanced through centralized training and distributed execution. Multi-machine obstacle avoidance actions generated based on ORCA are used as experience to accelerate algorithm convergence. This includes inputting observations into the ORCA algorithm to generate actions, inputting the actions into a commentator network to obtain reward values, and storing these as a whole in an experience pool. The overall flowchart of the deep reinforcement learning training acceleration process is as follows: Figure 4 As shown.

[0106] In summary, the method of this embodiment has at least the following advantages and beneficial effects compared to the prior art:

[0107] (1) This invention processes the point cloud data collected by lidar. First, it improves the Euclidean clustering algorithm based on dynamic thresholds, which greatly reduces clustering errors caused by distance. At the same time, it obtains the dynamic information of obstacles by relying on data association and extended Kalman filtering algorithm, which is used as the input of the obstacle avoidance algorithm. The advantage of this approach is that it does not significantly increase the execution time of the algorithm, and at the same time, it can greatly speed up the training of the algorithm compared to end-to-end training.

[0108] (2) This invention employs a deep reinforcement learning training acceleration method based on the ORCA algorithm to accelerate multi-UAV dynamic obstacle avoidance. In the early stage of the algorithm, the processed environmental information is input into the ORCA algorithm to obtain the output action, which is then combined with the obtained reward as a tuple and placed into the experience pool. This serves as successful obstacle avoidance experience, allowing the training process to quickly converge to a positive reward. Simultaneously, to address the conservative action selection problem of the ORCA algorithm, the later stage of the algorithm is exploration-driven, exploring a better path based on the path obtained in the early stage, thereby obtaining multi-UAV local path planning with short time and short distance. The entire process requires no human intervention during training, greatly accelerating the algorithm training process, and the trained path shows significant improvements in both time and path.

[0109] The present invention also provides a multi-UAV local dynamic obstacle avoidance device, comprising:

[0110] At least one processor;

[0111] At least one memory for storing at least one program;

[0112] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 5 The method shown.

[0113] This embodiment of the multi-UAV local dynamic obstacle avoidance device can execute the multi-UAV local dynamic obstacle avoidance method provided in the method embodiment of the present invention, and can execute any combination of the implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0114] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform... Figure 1 and Figure 5 The method shown.

[0115] This embodiment also provides a storage medium storing an executable file. Figure 1 and Figure 5 The instructions or program for a multi-UAV local dynamic obstacle avoidance method shown can execute any combination of implementation steps of the method embodiment when the instructions or program are run, and have the corresponding functions and beneficial effects of the method.

[0116] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0117] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0118] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0120] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0121] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0122] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0123] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0124] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for local dynamic obstacle avoidance for multiple unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: Formal modeling of distributed multi-UAV dynamic obstacle avoidance problem based on Markov decision process; Point cloud data is acquired by using lidar mounted on each drone, and obstacle information is obtained by processing the point cloud data. Construct a deep neural network to complete the mapping from observation input to action output; The observation input is the acquired obstacle information; The deep neural network is trained based on the ORCA algorithm, and the trained model is used for local dynamic obstacle avoidance of multiple UAVs. The obstacle information includes static obstacle information and dynamic obstacle information; Point cloud data is acquired by LiDAR mounted on each drone, and this data is processed to obtain obstacle information, including: Clustering is performed on the obtained point cloud data to identify reflection points belonging to the same obstacle within the point cloud data; After obtaining the clustering results, the center point and cluster radius of each cluster are obtained based on the coordinates of each point in each cluster, which are used as the position and size of the obstacle; Static obstacle information is obtained from single-frame point cloud data, and dynamic obstacle information is obtained by combining multi-frame point cloud data. The clustering process for the obtained point cloud data includes: Euclidean clustering algorithm is used to cluster point cloud data; the clustering criterion is whether the Euclidean distance between each point in space is less than a preset threshold. This preset threshold is determined in the following way: In the formula, , This represents the coordinates of the i-th point. , Indicates the coordinates of the lidar. For coefficients; The process of obtaining obstacle dynamic information by combining multi-frame point cloud data includes: Calculate the correlation degree between multiple frames of point cloud data, perform data association based on the correlation degree, obtain the position and velocity change values ​​as measurement values, input the measurement values ​​into the extended Kalman filter algorithm, and obtain obstacle dynamic information; The correlation degree is calculated as follows: In the formula, , , These represent the positional difference, magnitude difference, and reflection intensity difference between the two clusters in two frames, respectively. , , For coefficients; The training is based on a dual-delay deep deterministic policy gradient algorithm to generate action decisions in a continuous action space. The mapping from the observation space to the action space is accomplished by a deep neural network; during the training phase, the actions generated by the decision network are sampled after adding noise to explore a larger space; The training of deep neural networks based on the ORCA algorithm includes: Input environmental information into the ORCA algorithm to obtain the output action; The output action and reward are combined as a tuple and added to the experience pool as successful obstacle avoidance experience, so that the training process can quickly converge to a positive reward.

2. The multi-UAV local dynamic obstacle avoidance method according to claim 1, characterized in that, The formal modeling of the distributed multi-UAV dynamic obstacle avoidance problem based on Markov decision processes includes: The problem of distributed drones detecting obstacles, avoiding obstacles, and finally reaching the target point is modeled as a Markov decision process. The Markov decision process is represented using a six-tuple, i.e. ( , , , ,Ω, ),in This refers to the state space. This refers to the action space. This refers to the state transition model. This refers to the reward function, and Ω refers to the observation space. It is the observation probability distribution of the state of multiple unmanned aerial vehicle (UAV) systems.

3. The multi-UAV local dynamic obstacle avoidance method according to claim 2, characterized in that, For each drone in a multi-drone system, the drone observation space at time T is defined as follows: It contains the following part: the position of the i-th drone itself. The target point location of the i-th UAV The obtained observed environmental information was processed. The action space of a drone is the speed selected within the selectable speed space, where the selectable speed space is a continuous space, meaning the action design is... =[ , , ].

4. The multi-UAV local dynamic obstacle avoidance method according to claim 2, characterized in that, The tasks of the multi-UAV system include the following parts: Multiple drones should reach the target point as quickly as possible; Ensure navigation safety during the journey to the target point, that is, prevent collisions between drones or with obstacles; The reward function is designed as follows: in, The rewards include obstacle avoidance rewards. As a reward for the goal, The time reward is specifically expressed as follows: Obstacle Avoidance Rewards The negative reward is determined by the distance between the drone and the obstacle. Within a preset range, the closer the drone is to the obstacle, the greater the negative reward; the farther away it is, the smaller the reward. If a collision occurs, a fixed negative reward is given and the round ends. Target Rewards When the drone approaches the target point, it receives a positive reward. When it reaches the target point, it receives a fixed reward and the round ends. Conversely, when the drone moves away from the target point, it receives a negative reward proportional to the distance between the drone and the target point. Time reward The shorter the time it takes for the drone to reach the target point, the smaller the penalty.

5. A multi-UAV local dynamic obstacle avoidance device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-4.

6. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-4.

Images