Multi-sensor fusion unmanned aerial vehicle obstacle avoidance method and system

Abstract

The invention relates to a multi-sensor fusion unmanned aerial vehicle obstacle avoidance method and system. The method comprises the steps that a two-dimensional laser radar and a depth camera are used for obtaining a radar data sequence and point cloud data respectively; based on the point cloud data and the radar data sequence, respectively constructing a camera grid map and a radar grid map, and fusing the camera grid map and the radar grid map to obtain a grid map of the global environment; based on a grid map of a global environment, an initial path is generated by using a fast expansion random tree algorithm, and local optimization is performed on a part which is continuously turned for multiple times in the initial path by using an artificial field potential algorithm to obtain an optimized path. According to the method, the camera grid map and the radar grid map are respectively constructed by fusing the data of the two-dimensional laser radar and the depth camera, then the camera grid map and the radar grid map are fused to obtain the grid map of the global environment, the more accurate grid map of the global environment is generated by fusing the data of various sensors, and a more reliable basis is provided for obstacle avoidance of the unmanned aerial vehicle.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a multi-sensor fusion unmanned aerial vehicle obstacle avoidance method and system. Background Art

[0002] As a highly safe and low-cost aircraft, drones are widely used in low-altitude detection, search and rescue, military and other fields due to their high maneuverability and low cost. In order to ensure the flight safety and effectiveness of drones, people often need to monitor the flight status of drones at all times to ensure flight safety. Drones with a single sensor have poor autonomous obstacle avoidance effects and are prone to problems such as low accuracy and excessive information loss. Therefore, applying multi-sensor fusion to drone obstacle avoidance and improving the accuracy of autonomous obstacle avoidance will greatly reduce the cost of use and improve work efficiency.

[0003] The sensors used in the existing drone obstacle avoidance process are mainly:

[0004] (1) Laser radar: Laser radar uses laser as the signal source. The pulsed laser emitted by the laser hits the trees, roads, bridges and buildings on the ground, causing scattering. Part of the light wave will be reflected to the laser radar receiver. According to the laser ranging principle, the distance from the laser radar to the target point is calculated. The pulsed laser continuously scans the target object, and the data of all target points on the target object can be obtained. After imaging processing with this data, the target image can be obtained. Disadvantages of this sensor: a. It is greatly affected by weather and atmosphere during operation; b. The laser radar has a narrow beam and can only search for targets within a small range; c. The cost of use is high, and the size and installation location are relatively limited.

[0005] (2) Depth camera: Depth camera can be called 3D camera. The difference between it and ordinary 2D camera is that it can obtain the distance information from the object to the camera. In addition to the X and Y coordinates of the 2D plane, the three-dimensional coordinates of each point can be calculated, which can be used in scenes such as three-dimensional reconstruction, target positioning, navigation and obstacle avoidance. It is mainly divided into: structured light camera, stereo camera, and time-of-flight (TOF) camera. The disadvantages of this sensor include: a. It is very sensitive to ambient light. Light changes lead to large image deviation, it is not suitable for monotonous scenes without texture, the calculation complexity is high, and the baseline limits the measurement range; b. When the object being photographed has multiple edges and corners, the measurement result will be distorted, the light will cause the lens chip to quickly saturate and fail to detect the authenticity of infrared, resulting in distortion, and when multiple devices exist, they will interfere with each other, resulting in distortion.

[0006] At present, existing autonomous obstacle avoidance systems for drones mostly generate point cloud maps by simply adding up multi-sensor data. The generated point cloud maps have a large amount of data, are not compact enough, and require a lot of storage resources. In addition, it is impossible to determine whether the drone can pass smoothly between points in the point cloud map. The unstable point cloud quality and large amount of calculation restrict the application of multi-sensors in autonomous obstacle avoidance for drones. Summary of the invention

[0007] In view of this, it is necessary to provide a multi-sensor fusion UAV obstacle avoidance method and system to solve the above-mentioned defects of the prior art.

[0008] In order to solve the above problems, in a first aspect, an embodiment of the present invention provides a multi-sensor fusion unmanned aerial vehicle obstacle avoidance method, comprising:

[0009] Use two-dimensional laser radar and depth camera to obtain radar data sequence and point cloud data respectively;

[0010] Based on the point cloud data and the radar data sequence, a camera grid map and a radar grid map are constructed respectively, and the camera grid map and the radar grid map are fused to obtain a grid map of the global environment;

[0011] Based on the grid map of the global environment, the initial path is generated using the rapidly expanding random tree algorithm. For the parts of the initial path with multiple continuous turns, the artificial field potential algorithm is used for local optimization to obtain the optimized path.

[0012] For a road section consisting of multiple path points in the optimized path, a new road section is generated by connecting the starting path point and the ending path point of the road section, and it is determined whether there are obstacles on the new road section. According to the determination result, the path is dynamically adjusted.

[0013] Preferably, after respectively acquiring the radar data sequence and the point cloud data using the two-dimensional laser radar and the depth camera, the method further comprises:

[0014] Calibrate the internal parameters of the depth camera and calculate the external parameters of the depth camera relative to the lidar;

[0015] The point cloud data captured by the depth camera is downsampled and filtered to remove invalid point cloud information.

[0016] Preferably, based on the point cloud data and the radar data sequence, a camera grid map and a radar grid map are constructed respectively, including:

[0017] Based on the point cloud data, a three-dimensional dense point cloud map is constructed using an enhanced ORB-SLAM2 mapping algorithm;

[0018] Convert the three-dimensional dense point cloud map into an octree map, and construct a camera grid map using a map conversion tool;

[0019] A SLAM algorithm based on graph optimization is adopted to construct the radar grid map of the lidar using the radar data sequence.

[0020] Preferably, fusing the camera grid map and the radar grid map comprises:

[0021] According to the internal parameters of the depth camera and the external parameters of the depth camera relative to the lidar, the camera grid map and the radar grid map are fused to generate a grid map of the global environment.

[0022] Preferably, the part of the initial path with multiple consecutive turns is locally optimized using an artificial field potential algorithm to obtain an optimized path, specifically including:

[0023] Obtain a partial path with multiple consecutive turns in the initial path, select the starting point and the end point of the partial path, and use the artificial field potential algorithm for local optimization;

[0024] The new path generated by the artificial field potential algorithm is compared with the original path. If the number of turns in the new path is less than 60% of the original path, the new path is used to replace the original path.

[0025] Preferably, for a road section consisting of multiple path points in the optimized path, a new road section is generated by connecting the starting path point and the ending path point of the road section, and it is determined whether there is an obstacle on the new road section, and the path is dynamically adjusted according to the determination result, which specifically includes:

[0026] For a road segment consisting of no less than four path points in the optimized path, a new road segment is generated by taking the starting point and the ending point of the road segment as connection points.

[0027] Determine whether there are obstacles on the new road segment. If there are no obstacles, remove the middle point of the road segment to simplify the path;

[0028] If there are obstacles, the connection points are adjusted to generate a new section, and it is re-determined whether there are obstacles in the new section.

[0029] In a second aspect, an embodiment of the present invention provides a multi-sensor fusion unmanned aerial vehicle obstacle avoidance system, comprising:

[0030] An acquisition module, used to acquire radar data sequence and point cloud data using a two-dimensional laser radar and a depth camera respectively;

[0031] A map fusion module, used to construct a camera grid map and a radar grid map based on the point cloud data and the radar data sequence, respectively, and fuse the camera grid map and the radar grid map to obtain a grid map of the global environment;

[0032] The path optimization module is used to generate the initial path based on the grid map of the global environment using the fast expanding random tree algorithm. For the part of the initial path with multiple continuous turns, the artificial field potential algorithm is used to perform local optimization to obtain the optimized path.

[0033] The obstacle avoidance module is used to connect the starting path point and the ending path point of the section composed of multiple path points in the optimized path to generate a new section, and to determine whether there are obstacles on the new section, and dynamically adjust the path according to the determination result.

[0034] In a third aspect, the present invention further provides an electronic device, including a memory and a processor, wherein:

[0035] The memory is used to store programs;

[0036] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the multi-sensor fusion drone obstacle avoidance method as described in the embodiment of the first aspect of the present invention.

[0037] In a fourth aspect, the present invention further provides a computer-readable storage medium for storing computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the multi-sensor fusion drone obstacle avoidance method as described in the embodiment of the first aspect of the present invention.

[0038] Compared with the prior art, the multi-sensor fusion unmanned aerial vehicle obstacle avoidance method and system provided by the present invention has the following beneficial effects:

[0039] 1) The present invention fuses the data of two-dimensional lidar and depth camera to construct camera grid map and radar grid map respectively, and then fuses the two to obtain a grid map of the global environment. By utilizing the respective advantages and complementarity of lidar and depth camera, the accuracy of environmental perception is not only improved, but also the perception range is expanded.

[0040] 2) The present invention generates a more accurate global environment grid map by fusing data from multiple sensors, providing a more reliable basis for obstacle avoidance. At the same time, for the road sections in the optimized path, a new road section is generated by connecting the starting path point and the ending path point, and it is determined whether there are obstacles on the new road section, and the path is dynamically adjusted, thereby enhancing the real-time and stability of obstacle avoidance.

[0041] 3) The point cloud map of the prior art has a large amount of data, requires a lot of storage resources, and has high computational complexity. The present invention constructs a grid map to represent environmental information in a more compact way, greatly reducing the amount of data and storage requirements. At the same time, the rapid expansion random tree algorithm (RRT) is used to generate the initial path, and the artificial field potential algorithm is combined for local optimization, which improves the efficiency of path planning and reduces the amount of calculation.

[0042] 4) The present invention combines the ideas of global path planning and local path optimization, uses the RRT algorithm to generate the initial path, ensures the global feasibility of the path; then uses the artificial field potential algorithm to locally optimize the parts of the initial path with multiple continuous turns to obtain a smoother and more efficient path. This method not only ensures the global optimality of the path, but also improves the local quality of the path. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flow chart of the multi-sensor fusion unmanned aerial vehicle obstacle avoidance method provided by the present invention;

[0044] FIG. 2( a) is a radar grid map provided by the present invention;

[0045] FIG2( b ) is a camera grid map provided by the present invention;

[0046] FIG2( c ) is a grid map of the global environment provided by the present invention;

[0047] Figure 3 This is a structural block diagram of the multi-sensor fusion unmanned aerial vehicle obstacle avoidance system provided by the present invention;

[0048] Figure 4 This is a structural block diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0050] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0051] At present, existing autonomous obstacle avoidance systems for drones mostly generate point cloud maps by simply adding up multi-sensor data. The generated point cloud maps have a large amount of data, are not compact enough, and require a lot of storage resources. In addition, it is impossible to determine whether the drone can pass smoothly between points in the point cloud map. The unstable point cloud quality and large amount of calculation restrict the application of multi-sensors in autonomous obstacle avoidance for drones.

[0052] In view of this, the present invention provides a multi-sensor fusion unmanned aerial vehicle obstacle avoidance method, which fuses the data of a two-dimensional laser radar and a depth camera to construct a camera grid map and a radar grid map respectively, and then fuses the two to obtain a grid map of the global environment. By fusing the data of multiple sensors, a more accurate global environment grid map is generated, providing a more reliable basis for unmanned aerial vehicle obstacle avoidance. The following will be explained and introduced through multiple embodiments.

[0053] Figure 1 The flowchart of the multi-sensor fusion unmanned aerial vehicle obstacle avoidance method provided by the present invention is shown in FIG. Figure 1 The multi-sensor fusion unmanned aerial vehicle obstacle avoidance method provided by the present invention at least comprises the following steps:

[0054] Step S1, using a two-dimensional laser radar and a depth camera to obtain a radar data sequence and point cloud data respectively.

[0055] Specifically, the lidar and depth camera are used as environmental perception devices. The lidar scans the surrounding environment and generates a series of distance measurement values, which constitute the radar data sequence. The depth camera captures the depth information of the scene and generates point cloud data containing three-dimensional coordinates.

[0056] It is understandable that the internal parameters of the depth camera (such as focal length, optical center position, distortion coefficient, etc.) are crucial to its measurement accuracy. Therefore, before using the depth camera, it needs to be accurately calibrated to ensure the accuracy of subsequent processing.

[0057] In this embodiment, the internal parameters of the depth camera are calibrated, and the external parameters of the depth camera relative to the laser radar are calculated. The external parameters refer to the position and posture (i.e., rotation and translation) of the depth camera relative to the laser radar. In order to fuse the data obtained by both into a unified coordinate system, these external parameters need to be calculated. For example, a specific calibration tool or mode can be used to capture the data of the laser radar and the depth camera at the same time, and the relative position and posture between them can be calculated by an algorithm.

[0058] The point cloud data captured by the depth camera may contain a large amount of invalid information, such as point clouds that are too high or near the ground. These point clouds may be unnecessary for subsequent map construction and path planning, and may even introduce errors. Therefore, this embodiment downsamples and filters the point cloud data captured by the depth camera to remove invalid point cloud information. Downsampling can reduce the number of point clouds and reduce computational complexity; filtering can remove specific point clouds (such as point clouds near the ground) to improve the accuracy and reliability of the data.

[0059] Step S2: constructing a camera grid map and a radar grid map based on the point cloud data and the radar data sequence, respectively, and fusing the camera grid map and the radar grid map to obtain a grid map of the global environment.

[0060] Step S2 may specifically include the following steps:

[0061] S21, constructing a camera grid map based on the point cloud data.

[0062] Specifically, the point cloud data captured by the depth camera is processed by the enhanced ORB-SLAM2 (Oriented FAST and Rotated BRIEF-SLAM 2) algorithm. ORB-SLAM2 is a feature point-based visual SLAM algorithm that can estimate the trajectory of the camera in real time and build a sparse point cloud map. The "enhanced" here means that the algorithm is extended to support the construction of three-dimensional dense point cloud maps.

[0063] Although the three-dimensional dense point cloud map is rich in information, it is very time-consuming and resource-intensive to process. Therefore, this embodiment uses an octree tool to convert the dense point cloud map into a more efficient data structure, namely an octree map. An octree is a tree-shaped data structure that recursively partitions space and can efficiently represent objects in three-dimensional space. With an octree map, spatial information can be queried and processed more quickly.

[0064] Next, a map conversion tool (such as Octomap) is used to further convert the octree map into a two-dimensional grid map to obtain a camera grid map. Figure 2(a) is a radar grid map provided by the present invention. The grid map is a map representation method that divides the space into a series of small cells (grids), each cell representing a specific area in the environment. The map represents obstacles and free space in the environment in a simple grid form.

[0065] It is understandable that the point cloud map of the prior art has a large amount of data, consumes a lot of storage resources, and has high computational complexity. The present invention constructs a grid map to represent environmental information in a more compact manner, greatly reducing the amount of data and storage requirements.

[0066] S22: constructing a radar grid map based on the radar data sequence.

[0067] Specifically, a SLAM algorithm based on graph optimization is used to construct a radar grid map of the lidar using radar data sequences. Among them, the SLAM algorithm based on graph optimization is a specific SLAM implementation method, which constructs a graph to represent the robot's position (nodes) at different time points and the relative relationship (edges) between these positions. Then, an optimization algorithm is used to adjust the positions of these nodes to minimize the error between them, thereby obtaining more accurate maps and positioning information. First, the radar data is converted into obstacle position information in the environment; then, the position of the obstacle is marked in the grid map based on this information; finally, the map is gradually improved by continuously updating and integrating new radar data.

[0068] In this embodiment, a SLAM algorithm based on graph optimization is used to process the data sequence obtained by the laser radar, and by continuously updating and fusing these data, a radar grid map containing the location information of obstacles in the environment is finally constructed. Figure 2(b) is a camera grid map provided by the present invention.

[0069] S23, fusing the camera grid map and the radar grid map.

[0070] Specifically, the two two-dimensional grid maps constructed separately are fused according to the internal and external parameters of the depth camera and the laser radar calculated in the above-mentioned embodiment. In the map fusion process, the two maps are aligned to the same coordinate system, and the data from different sensors are fused according to the Bayesian estimation rule.

[0071] By fusing the camera grid map and the radar grid map, a grid map of the global environment is obtained. FIG2(c) is a grid map of the global environment provided by the present invention, which integrates the data of the lidar and the depth camera to provide more accurate and comprehensive environmental information. The generated global environment grid map is used as the input of the path planning algorithm. The path planning algorithm (such as RRT and artificial field potential algorithm) generates the flight path of the drone based on the information in the map and takes into account obstacles to avoid collisions.

[0072] The present invention fuses the data of two-dimensional lidar and depth camera to construct camera grid map and radar grid map respectively, and then fuses the two to obtain a grid map of the global environment. By utilizing the respective advantages and complementarity of lidar and depth camera, the accuracy of environmental perception is not only improved, but also the perception range is expanded.

[0073] The present invention generates a more accurate global environment grid map by fusing data from multiple sensors, providing a more reliable basis for obstacle avoidance.

[0074] Step S3, based on the grid map of the global environment, use the rapidly expanding random tree algorithm to generate an initial path, and use the artificial field potential algorithm to perform local optimization on the part of the initial path with multiple continuous turns to obtain an optimized path.

[0075] It is understandable that in path planning, a path with too many consecutive turns often means that the path is not smooth enough, which may cause the drone to frequently adjust its direction during flight, increasing flight difficulty and energy consumption. Therefore, finding these partial paths with too many turns and optimizing them using the artificial field potential algorithm can make the path smoother and improve the flight efficiency and stability of the drone.

[0076] Specifically, the initial flight path of the UAV is generated using the Rapidly Expanding Random Tree (RRT) algorithm. RRT is a path planning algorithm based on random sampling, which can quickly generate a feasible path from the starting point to the end point in a high-dimensional space. This method is suitable for global path planning and can provide a preliminary flight route for the UAV in a large area.

[0077] In the initial path, there may be some parts with multiple continuous turns, which are usually not optimal because they may increase the complexity and time cost of the flight. In order to optimize these parts, the present invention adopts an artificial field potential algorithm. The artificial field potential algorithm is an algorithm that simulates the laws of particle motion in a physical field (such as an electric field or a magnetic field). In path planning, it can be used to guide the drone to fly along a smoother and more direct path. By calculating the potential field value of each point on the path and adjusting the flight direction of the drone according to the potential field value, local optimization of the path can be achieved.

[0078] In this embodiment, a partial path with a continuous turning number greater than 3 in the initial path is obtained. These parts are generally considered to be areas where the path is not smooth enough or the efficiency is low. The starting point and the end point of the partial path are selected, and the artificial field potential algorithm is used for local optimization. The new path generated by the artificial field potential algorithm is compared with the original path. If the number of turns in the new path is less than 60% of the original path, the new path is used to replace the original path. This ensures continuous optimization and improvement of the path.

[0079] By combining the initial path generated by the RRT algorithm with the local optimization of the artificial field potential algorithm, the present invention can obtain a smoother, more direct and feasible flight path. This optimized path not only takes into account the obstacle distribution of the global environment, but also reduces unnecessary turns and flight distance through local optimization, thereby improving the flight efficiency and safety of the UAV.

[0080] Step S4, for the road section composed of multiple path points in the optimized path, the starting path point and the ending path point of the road section are connected to generate a new road section, and it is determined whether there are obstacles on the new road section, and the path is dynamically adjusted according to the determination result.

[0081] Specifically, for a road segment consisting of no less than four path points in the optimized path, a new road segment is generated with the starting point and the ending point of the road segment as connection points. It is determined whether there is an obstacle on the new road segment. If there is no obstacle, the middle point of the road segment is removed to simplify the path. If there is an obstacle, the connection point is adjusted to generate a new road segment, and it is re-determined whether there is an obstacle on the new road segment.

[0082] In practical applications, a segment consisting of four path points is taken as an example. A segment consisting of four consecutive points is selected for investigation. These four points are marked as the first point, the second point, the third point, and the fourth point. First, try to directly connect the first and fourth points of the four points to form a new straight line segment path. The purpose of this step is to check whether the intermediate points (i.e., the second and third points) can be directly skipped to simplify the path. Next, check whether there are obstacles on this newly generated straight line segment path. This is achieved by comparing this path with known obstacle locations or map data. If there are no obstacles on this new path, it means that the first and fourth points can be directly connected without passing through the second and third points. Therefore, the second and third points can be deleted to simplify the path. If there are obstacles on the new path, the second and third points cannot be simply deleted. At this time, try to connect the first and third points to form a new path segment. Next, check again whether there are obstacles on this new path segment. If there are no obstacles, it means that the first and third points can be directly connected. At this time, the second point can be deleted, but the third point can be retained as part of the path. If there are still obstacles, it means that the path cannot be further simplified. At this time, the current judgment loop should be exited and other possible path optimization strategies should be considered or the original path should be kept unchanged. The above process continues, and four consecutive points are selected for inspection and optimization each time until the end of the path (that is, "the final point is the fourth point"). In this way, through continuous iteration and optimization, an optimal path that is both simple and safe can be generated.

[0083] During the flight of the drone, the above path optimization process will continue to ensure that the drone can dynamically adjust its flight path according to real-time environmental information, so as to effectively avoid obstacles and reach the target location.

[0084] Figure 3 This is a structural block diagram of the multi-sensor fusion unmanned aerial vehicle obstacle avoidance system provided by the present invention, refer to Figure 3 , the multi-sensor fusion UAV obstacle avoidance system 300 includes:

[0085] An acquisition module 301 is used to acquire radar data sequence and point cloud data using a two-dimensional laser radar and a depth camera respectively;

[0086] A map fusion module 302 is used to construct a camera grid map and a radar grid map based on the point cloud data and the radar data sequence, respectively, and fuse the camera grid map and the radar grid map to obtain a grid map of the global environment;

[0087] The path optimization module 303 is used to generate an initial path based on a grid map of the global environment using a rapidly expanding random tree algorithm, and to perform local optimization using an artificial field potential algorithm for a portion of the initial path with multiple consecutive turns to obtain an optimized path;

[0088] The obstacle avoidance module 304 is used to connect the starting path point and the ending path point of the section composed of multiple path points in the optimized path to generate a new section, and determine whether there are obstacles on the new section, and dynamically adjust the path according to the determination result.

[0089] The multi-sensor fusion UAV obstacle avoidance system provided by the present invention executes the multi-sensor fusion UAV obstacle avoidance method provided by the aforementioned embodiments through the above-mentioned modules. The multi-sensor fusion UAV obstacle avoidance method has been described in detail in the above-mentioned embodiments, and will not be repeated in this embodiment.

[0090] Compared with the prior art, the multi-sensor fusion unmanned aerial vehicle obstacle avoidance method and system provided by the present invention has the following beneficial effects:

[0091] 1) The present invention fuses the data of two-dimensional lidar and depth camera to construct camera grid map and radar grid map respectively, and then fuses the two to obtain a grid map of the global environment. By utilizing the respective advantages and complementarity of lidar and depth camera, the accuracy of environmental perception is not only improved, but also the perception range is expanded.

[0092] 2) The present invention generates a more accurate global environment grid map by fusing data from multiple sensors, providing a more reliable basis for obstacle avoidance. At the same time, for the road sections in the optimized path, a new road section is generated by connecting the starting path point and the ending path point, and it is determined whether there are obstacles on the new road section, and the path is dynamically adjusted, thereby enhancing the real-time and stability of obstacle avoidance.

[0093] 3) The point cloud map of the prior art has a large amount of data, requires a lot of storage resources, and has high computational complexity. The present invention constructs a grid map to represent environmental information in a more compact way, greatly reducing the amount of data and storage requirements. At the same time, the rapid expansion random tree algorithm (RRT) is used to generate the initial path, and the artificial field potential algorithm is combined for local optimization, which improves the efficiency of path planning and reduces the amount of calculation.

[0094] 4) The present invention combines the ideas of global path planning and local path optimization, uses the RRT algorithm to generate the initial path, ensures the global feasibility of the path; then uses the artificial field potential algorithm to locally optimize the parts of the initial path with multiple continuous turns to obtain a smoother and more efficient path. This method not only ensures the global optimality of the path, but also improves the local quality of the path.

[0095] Figure 4 The structural block diagram of the electronic device provided by the present invention is as follows: Figure 4 As shown, the present invention also provides an electronic device, and the electronic device 400 can be a computing device such as a mobile terminal, a desktop computer, a notebook, a PDA, and a server. The electronic device 400 includes a processor 401 and a memory 402, wherein the memory 402 stores a multi-sensor fusion drone obstacle avoidance program 403.

[0096] In some embodiments, the memory 402 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 402 may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Furthermore, the memory 402 may also include both an internal storage unit of the computer device and an external storage device. The memory 402 is used to store application software and various types of data installed on the computer device, such as program codes for installing the computer device. The memory 402 can also be used to temporarily store data that has been output or is to be output. In one embodiment, when the multi-sensor fusion drone obstacle avoidance program 403 is executed by the processor 401, the following steps are implemented:

[0097] Use two-dimensional laser radar and depth camera to obtain radar data sequence and point cloud data respectively;

[0098] Based on the point cloud data and the radar data sequence, a camera grid map and a radar grid map are constructed respectively, and the camera grid map and the radar grid map are fused to obtain a grid map of the global environment;

[0099] Based on the grid map of the global environment, the initial path is generated using the rapidly expanding random tree algorithm. For the parts of the initial path with multiple continuous turns, the artificial field potential algorithm is used for local optimization to obtain the optimized path.

[0100] For a road section consisting of multiple path points in the optimized path, a new road section is generated by connecting the starting path point and the ending path point of the road section, and it is determined whether there are obstacles on the new road section. According to the determination result, the path is dynamically adjusted.

[0101] In some embodiments, the processor 401 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 402, such as executing a multi-sensor fusion drone obstacle avoidance program.

[0102] This embodiment also provides a computer-readable storage medium, on which a multi-sensor fusion drone obstacle avoidance program is stored. When the multi-sensor fusion drone obstacle avoidance program is executed by a processor, the following steps are implemented:

[0103] Use two-dimensional laser radar and depth camera to obtain radar data sequence and point cloud data respectively;

[0104] Based on the point cloud data and the radar data sequence, a camera grid map and a radar grid map are constructed respectively, and the camera grid map and the radar grid map are fused to obtain a grid map of the global environment;

[0105] Based on the grid map of the global environment, the initial path is generated using the rapidly expanding random tree algorithm. For the parts of the initial path with multiple continuous turns, the artificial field potential algorithm is used for local optimization to obtain the optimized path.

[0106] For a road section consisting of multiple path points in the optimized path, a new road section is generated by connecting the starting path point and the ending path point of the road section, and it is determined whether there are obstacles on the new road section. According to the determination result, the path is dynamically adjusted.

[0107] The above-described embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-sensor fusion unmanned aerial vehicle obstacle avoidance method, characterized in that: include: Use two-dimensional laser radar and depth camera to obtain radar data sequence and point cloud data respectively; Based on the point cloud data and the radar data sequence, a camera grid map and a radar grid map are constructed respectively, and the camera grid map and the radar grid map are fused to obtain a grid map of the global environment; Based on the grid map of the global environment, the initial path is generated using the rapidly expanding random tree algorithm. For the parts of the initial path with multiple continuous turns, the artificial field potential algorithm is used for local optimization to obtain the optimized path. For a road section consisting of multiple path points in the optimized path, a new road section is generated by connecting the starting path point and the ending path point of the road section, and it is determined whether there are obstacles on the new road section. According to the determination result, the path is dynamically adjusted.

2. The multi-sensor fusion unmanned aerial vehicle obstacle avoidance method according to claim 1, characterized in that: After respectively acquiring radar data sequence and point cloud data by using a two-dimensional laser radar and a depth camera, the method further includes: Calibrate the internal parameters of the depth camera and calculate the external parameters of the depth camera relative to the lidar; The point cloud data captured by the depth camera is downsampled and filtered to remove invalid point cloud information.

3. The multi-sensor fusion unmanned aerial vehicle obstacle avoidance method according to claim 1, characterized in that: Based on the point cloud data and the radar data sequence, a camera grid map and a radar grid map are constructed respectively, including: Based on the point cloud data, a three-dimensional dense point cloud map is constructed using an enhanced ORB-SLAM2 mapping algorithm; Convert the three-dimensional dense point cloud map into an octree map, and construct a camera grid map using a map conversion tool; A SLAM algorithm based on graph optimization is adopted to construct the radar grid map of the lidar using the radar data sequence.

4. The multi-sensor fusion unmanned aerial vehicle obstacle avoidance method according to claim 2, characterized in that: The camera grid map and the radar grid map are fused, including: According to the internal parameters of the depth camera and the external parameters of the depth camera relative to the lidar, the camera grid map and the radar grid map are fused to generate a grid map of the global environment.

5. The multi-sensor fusion unmanned aerial vehicle obstacle avoidance method according to claim 1, characterized in that: The artificial field potential algorithm is used to perform local optimization on the portion of the initial path that makes multiple continuous turns to obtain an optimized path, specifically including: Obtain a partial path with multiple consecutive turns in the initial path, select the starting point and the end point of the partial path, and use the artificial field potential algorithm for local optimization; The new path generated by the artificial field potential algorithm is compared with the original path. If the number of turns in the new path is less than 60% of the original path, the new path is used to replace the original path.

6. The multi-sensor fusion unmanned aerial vehicle obstacle avoidance method according to claim 1, characterized in that: For a road segment consisting of multiple path points in the optimized path, a new road segment is generated by connecting the starting path point and the ending path point of the road segment, and it is determined whether there is an obstacle on the new road segment. According to the determination result, the path is dynamically adjusted, specifically including: For a road segment consisting of no less than four path points in the optimized path, a new road segment is generated by taking the starting point and the ending point of the road segment as connection points. Determine whether there are obstacles on the new road segment. If there are no obstacles, remove the middle point of the road segment to simplify the path; If there are obstacles, the connection points are adjusted to generate a new section, and it is re-determined whether there are obstacles in the new section.

7. A multi-sensor fusion unmanned aerial vehicle obstacle avoidance system applied to the multi-sensor fusion unmanned aerial vehicle obstacle avoidance method according to any one of claims 1 to 6, characterized in that: include: An acquisition module, used to acquire radar data sequence and point cloud data using a two-dimensional laser radar and a depth camera respectively; A map fusion module, used to construct a camera grid map and a radar grid map based on the point cloud data and the radar data sequence, respectively, and fuse the camera grid map and the radar grid map to obtain a grid map of the global environment; The path optimization module is used to generate the initial path based on the grid map of the global environment using the fast expanding random tree algorithm. For the part of the initial path with multiple continuous turns, the artificial field potential algorithm is used to perform local optimization to obtain the optimized path. The obstacle avoidance module is used to connect the starting path point and the ending path point of the section composed of multiple path points in the optimized path to generate a new section, and to determine whether there are obstacles on the new section, and dynamically adjust the path according to the determination result.

8. An electronic device, It is characterized in that comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the multi-sensor fusion drone obstacle avoidance method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the multi-sensor fusion drone obstacle avoidance method described in any one of claims 1 to 6.

Images