A UAV with automatic switching navigation system

Abstract

The present invention discloses an unmanned aerial vehicle (UAV) with an automatically switching navigation system. The UAV includes a UAV body, a laser SLAM system for implementing the UAV's navigation, a visual SLAM system for assisting the UAV's navigation, and a controller. When the controller detects that the error between the position and pose obtained by a laser inertial odometry in the laser SLAM system and a reference true value exceeds a switching threshold, the laser SLAM system switches to the visual SLAM system for UAV navigation. The UAV provided by the present invention can still make motion decisions in complex environments even when encountering inclement weather such as heavy rain, thick smoke, or dense fog, thereby improving the accuracy and stability of the UAV, and is easy to use while also enhancing safety.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicles (UAVs), and in particular relates to an UAV with an automatic switching navigation system. Background Art

[0002] Due to the advantages of lidar, such as high precision, wide viewing angle, and wide ranging range, some existing drones currently use lidar SLAM technology for navigation. However, the applicant's research found that the attenuation of lidar lasers is small in clear weather and the propagation distance is long. However, in bad weather such as heavy rain, thick smoke, and dense fog, the attenuation increases sharply, significantly affecting the laser's propagation distance. Therefore, when a drone using lidar technology encounters heavy rain, thick smoke, or dense fog, the data obtained by the lidar will be inaccurate, resulting in the drone being unable to determine its own position and attitude, which in turn makes it difficult for the drone to understand the complex environment and make movement decisions.

[0003] Therefore, how to enable drones to make movement decisions in complex environments when encountering severe weather such as heavy rain, thick smoke or fog is a technical problem that needs to be solved urgently. Summary of the Invention

[0004] An embodiment of the present invention provides a drone with an automatically switchable navigation system. By switching between different navigation systems, the drone can still make movement decisions in complex environments when encountering severe weather such as heavy rain, thick smoke or dense fog.

[0005] Based on this, an embodiment of the present invention provides a drone with an automatic switching navigation system, including a drone body, a laser SLAM system for realizing the navigation of the drone, a visual SLAM system for assisting the navigation of the drone, and a controller; when the controller detects that the error between the posture obtained by the laser inertial odometer in the laser SLAM system and the reference true value is greater than the switching set threshold, the laser SLAM system is switched to the visual SLAM system for drone navigation.

[0006] In the above possible implementation, the laser SLAM system includes a laser radar unit and an IMU inertial unit. The laser SLAM system uses the laser radar in combination with the inertial odometer in the IMU inertial unit to calibrate the odometer's posture information in real time.

[0007] In the above possible implementation method, the laser SLAM system uses the scan matching algorithm of the lidar point cloud data to calculate the posture of the drone and uses it as the reference true value of the inertial odometry posture; when the error between the posture obtained by the inertial odometry and the reference true value is greater than a first set threshold, a calibration operation is performed and the posture after calibration is used to update the original posture of the inertial odometry; otherwise, no calibration is performed.

[0008] In the above possible implementation, the scan matching algorithm using lidar data is used to calculate the drone's posture and serve as the reference true value of the inertial odometry posture, including: continuously acquiring environmental scanning information provided by the lidar, matching the two frames of point cloud data, calculating the drone displacement between two consecutive frames of laser point cloud data, calculating the drone displacement between all laser frames, and combining the drone's starting posture to calculate the real-time posture of the drone, and the real-time posture serves as the reference true value of the inertial odometry posture.

[0009] In the above possible implementation, the visual SLAM system includes a binocular camera unit located on the front of the drone body, and the visual SLAM system uses a feature point depth acquisition method of triangulation of multiple keyframe matching points to reconstruct the three-dimensional position of the feature points.

[0010] In the above possible implementation, the method for acquiring feature point depth by triangulating multiple keyframe matching points includes:

[0011] A six-dimensional vector multi-keyframe matching point triangulation feature point depth acquisition method is used to represent the three-dimensional position of an environmental feature point, including the position of the UAV's binocular camera unit and the direction and depth information of the environmental feature point where the UAV is located;

[0012] The environment is continuously updated. When the feature estimation covariance is less than the second set threshold, the six-dimensional representation of the environmental feature points of the drone is converted into three-dimensional Euclidean coordinates. Multiple feature points are established in the same frame image, and the representation of the feature points belonging to the same frame is reduced to the form of a binocular camera unit posture plus multiple depths, so as to effectively reduce the length of the drone system state.

[0013] In the above possible implementation, the laser radar unit and the IMU inertial unit are integrated into a frame, the laser radar unit is located in front of the drone body, and the binocular camera unit is located above the laser radar.

[0014] In the above possible implementation methods, the laser SLAM system also includes a laser system map update module and a global map fusion module. The laser system map update module is used to update the map information therein according to the drone posture information updated in real time by the laser SLAM system, and the global map fusion module fuses the local map information of all local laser SLAM systems to generate a global three-dimensional map.

[0015] In the above possible implementation, the visual SLAM system further includes a visual system map update module, which updates the map information therein according to the drone pose information updated in real time by the visual SLAM system.

[0016] The above possible implementation methods also include a multi-sensor fusion module, which fuses the drone pose motion information obtained by the laser SLAM system, the drone pose motion information obtained by the visual SLAM system, and the updated global three-dimensional map information to obtain optimized drone pose motion information; when there are external GPS and / or RTK signals, the GPS and / or RTK information are simultaneously integrated to obtain the drone pose motion information with the earth coordinate system.

[0017] From the above, the above embodiment of the present invention provides a drone with automatic switching navigation system, which is based on the combination of a visual SLAM system and a three-dimensional laser SLAM system. When the navigation positioning of one of the systems deviates, it can be seamlessly switched to another system by switching, thereby improving the environmental adaptability and stability of the drone. At the same time, the laser SLAM system calibrates and corrects the inertial odometer in real time, making the system positioning and navigation more accurate and applicable to a wider range of scenarios.

[0018] The beneficial effects of the present invention are as follows:

[0019] 1. When the drone encounters severe weather such as heavy rain, thick smoke or fog, it can still make movement decisions in complex environments, which improves the accuracy and stability of the drone, makes it easier to use, and improves safety.

[0020] 2. By switching between the two systems, it is possible to prevent the entire drone from stopping operating when one of the systems fails.

[0021] 3. It can be flown indoors or in any condition where there is no satellite GPS signal reception. It is safe and reliable, not easily interfered with by wireless signals, and can fly autonomously without any communication connection.

[0022] 4. Achieve high-reliability hovering and autonomous flight of UAVs, and independently plan routes quickly and fly to the destination.

[0023] 5. The system's lidar is a self-luminous sensor that can fly in complete darkness. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 Schematic diagram of the structure of a drone with an automatic switching navigation system in an embodiment of the present invention.

[0026] Figure 2 2 is a functional module diagram of a drone with an automatic switching navigation system in an embodiment of the present invention.

[0027] Figure 3 A switching flow chart for automatically switching a navigation system drone provided by an embodiment of the present invention.

[0028] Figure 4 This is a general flow chart of the method for visual fusion lidar SLAM positioning and navigation of a drone in an embodiment of the present invention.

[0029] Figure 5 This is a flowchart of the operation of the three-dimensional laser SLAM system in an embodiment of the present invention.

[0030] Figure 6 This is a flowchart of the operation of the visual SLAM system in an embodiment of the present invention.

[0031] Figure 7 This is a UAV system module based on vision fusion 3D lidar SLAM positioning and navigation in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0033] The words "first, second, third, etc." or module A, module B, module C and other similar terms in the specification and claims are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that the specific order or sequence can be interchanged where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] In the following description, the reference to step numbers, such as S110, S120, etc., does not necessarily imply that the steps must be executed in that order. The order of the steps may be interchanged or performed simultaneously, as appropriate. The term "comprising" used in the specification and claims should not be construed as limiting the list thereafter; it does not exclude other elements or steps. Therefore, it should be interpreted as specifying the presence of the features, integers, steps, or components mentioned, but does not preclude the presence or addition of one or more other features, integers, steps, or components, or groups thereof. Therefore, the expression "a device comprising means A and B" should not be limited to a device consisting solely of components A and B. References to "one embodiment" or "an embodiment" throughout this specification mean that the particular features, structures, or characteristics described in connection with that embodiment are included in at least one embodiment of the present application. Therefore, appearances of the phrases "in one embodiment" or "in an embodiment" throughout this specification do not necessarily refer to the same embodiment, but may do so. Furthermore, in one or more embodiments, the particular features, structures, or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0036] See also Figure 1 、 Figure 2 、 Figure 3 The embodiment of the present invention provides a drone with an automatic switching navigation system, which includes: a drone body (unnumbered), a laser SLAM system 100 for realizing the navigation of the drone, a visual SLAM system 200 for assisting the navigation of the drone, and a controller 600; when the controller 600 detects that the error between the posture obtained by the laser inertial odometer in the laser SLAM system 100 and the reference true value is greater than the switching set threshold, the laser SLAM system 100 is switched to the visual SLAM system 200 for drone navigation.

[0037] In the above possible implementation, the laser SLAM system 100 includes a laser radar unit and an IMU inertial unit. The laser SLAM system 100 uses the laser radar in combination with the inertial odometer in the IMU inertial unit to calibrate the odometer's posture information in real time.

[0038] In the above possible implementation method, the laser SLAM system 100 uses the scan matching algorithm of the laser radar point cloud data to calculate the posture of the drone and uses it as the reference true value of the inertial odometry posture; when the error between the posture obtained by the inertial odometry and the reference true value is greater than a first set threshold, a calibration operation is performed and the posture after calibration is used to update the original posture of the inertial odometry; otherwise, no calibration is performed.

[0039] In the above possible implementation, the scan matching algorithm using lidar data is used to calculate the drone's posture and serve as the reference true value of the inertial odometry posture, including: continuously acquiring environmental scanning information provided by the lidar, matching the two frames of point cloud data, calculating the drone displacement between two consecutive frames of laser point cloud data, calculating the drone displacement between all laser frames, and combining the drone's starting posture to calculate the real-time posture of the drone, and the real-time posture serves as the reference true value of the inertial odometry posture.

[0040] In the above possible implementation, the visual SLAM system 200 includes a binocular camera unit located on the front of the drone body, and the visual SLAM system 200 uses a feature point depth acquisition method of triangulation of multiple keyframe matching points to reconstruct the three-dimensional position of the feature points.

[0041] In the above possible implementation, the method for acquiring feature point depth by triangulating multiple keyframe matching points includes:

[0042] A six-dimensional vector multi-keyframe matching point triangulation feature point depth acquisition method is used to represent the three-dimensional position of an environmental feature point, including the position of the UAV's binocular camera unit and the direction and depth information of the environmental feature point where the UAV is located;

[0043] The environment is continuously updated. When the feature estimation covariance is less than the second set threshold, the six-dimensional representation of the environmental feature points of the drone is converted into three-dimensional Euclidean coordinates. Multiple feature points are established in the same frame image, and the representation of the feature points belonging to the same frame is reduced to the form of a binocular camera unit posture plus multiple depths, so as to effectively reduce the length of the drone system state.

[0044] In the above possible implementation, the laser radar unit and the IMU inertial unit are integrated into a frame, the laser radar unit is located in front of the drone body, and the binocular camera unit is located above the laser radar.

[0045] In the above possible implementation methods, the laser SLAM system 100 also includes a laser system map update module and a global map fusion module. The laser system map update module is used to update the map information therein according to the drone posture information updated in real time by the laser SLAM system 100, and the global map fusion module fuses the local map information of all local laser SLAM systems to generate a global three-dimensional map.

[0046] In the above possible implementations, the visual SLAM system 200 further includes a visual system map update module, which updates the map information therein according to the drone pose information updated in real time by the visual SLAM system 200.

[0047] The above possible implementation methods also include a multi-sensor fusion module, which fuses the drone pose motion information obtained by the laser SLAM system, the drone pose motion information obtained by the visual SLAM system, and the updated global three-dimensional map information to obtain optimized drone pose motion information; when there are external GPS and / or RTK signals, the GPS and / or RTK information are simultaneously integrated to obtain the drone pose motion information with the earth coordinate system.

[0048] In the above-mentioned embodiment of the present invention, the drone that automatically switches the navigation system uses a three-dimensional laser SLAM system for navigation when flying at the beginning or in a normal environment. When the controller detects that the error between the posture obtained by the odometer in the three-dimensional laser SLAM system and the reference true value is greater than the preset switching threshold, the navigation system is switched to the visual SLAM system to guide the drone to continue navigation flight.

[0049] The following further lists embodiments for implementing the present invention in more detail.

[0050] On the one hand, see Figure 4-Figure 6 The embodiment of the present invention provides a method for visual fusion laser radar SLAM positioning and navigation for unmanned aerial vehicle positioning and navigation, comprising the following steps:

[0051] (1) Obtain the first fusion pose and motion information generated by the 3D laser SLAM system;

[0052] (2) Obtain the second fusion pose and motion information generated by the visual SLAM system;

[0053] (3) comparing the first fused pose and motion information with the second fused pose and motion information to obtain a third comparison error, and comparing the third comparison error with the system calibration error;

[0054] (4) When the third comparison error is greater than the system calibration error, switch to the visual SLAM system for navigation of the UAV.

[0055] In the above embodiment, further comprising:

[0056] (5) In step (4), further: when the third comparison error is greater than the system calibration error, switch to the visual SLAM system to obtain the third fused pose and motion information for the UAV positioning and navigation; or when the third comparison error is less than or equal to the system calibration error, obtain the second fused pose and motion information and continue to use the laser SLAM system for the UAV positioning and navigation;

[0057] (6) Obtaining map update information of the 3D laser SLAM system and / or obtaining map update information of the visual SLAM system;

[0058] (7) Obtaining the location information specified by the user, and generating the best route based on the location information specified by the user, the map update information, the third fusion pose and motion information or the second fusion pose and motion information, performing real-time positioning and autonomous navigation of the UAV, and displaying the location of the UAV and controlling the flight of the UAV;

[0059] In the embodiment of the present invention, the system calibration error is a system preset value.

[0060] Furthermore, in an embodiment of the present invention, the step (1) of obtaining the first fusion pose and motion information generated by the three-dimensional laser SLAM system further includes the following steps:

[0061] (1-1) Obtain the first IMU pose and motion information of the current frame collected by the first IMU unit, and obtain the point cloud data information of the current frame collected by the three-dimensional laser radar;

[0062] (1-2), filtering and preprocessing the collected point cloud data;

[0063] (1-3) Solve the filtered point cloud data to obtain the lidar pose and motion information of the current frame and the local map of the surrounding environment of the current frame in the laser system, and update the map in the laser system;

[0064] (1-4), fuse all local maps to generate a global map;

[0065] (1-5) combining the lidar pose and motion information of the previous frame with the lidar pose and motion information of the current frame to generate a first motion trajectory;

[0066] (1-6), combining the first IMU pose and motion information of the previous frame with the first IMU pose and motion information of the current frame to generate a second motion trajectory;

[0067] (1-7), comparing the first motion trajectory and the second motion trajectory to obtain a first comparison error, and comparing the first comparison error with a laser SLAM calibration error, wherein the laser SLAM calibration error is a system preset value;

[0068] (1-8) According to the comparison result of the first comparison error and the laser SLAM calibration error, the position and motion information of the first IMU and the laser radar are fused, and combined with the currently updated global map model to obtain the first fused position and motion information of the current frame;

[0069] (1-9), output the first fused pose and motion information and global map information.

[0070] Furthermore, in the embodiment of the present invention, the map update information of the three-dimensional laser SLAM system is obtained in step (6) by solving the point cloud data after filtering, and the local map information of the surrounding environment of the current frame in the laser system is updated according to the local map information.

[0071] Furthermore, the embodiment of the present invention also includes step (1-8): fusing all local map information to generate a global map.

[0072] Furthermore, in the embodiment of the present invention, in step (1-7), obtaining the first fused pose and motion information of the current frame includes outputting the first fused pose and motion information in combination with the currently updated global map information;

[0073] Furthermore, in an embodiment of the present invention, the filtering and preprocessing of the collected point cloud data in step (1-2) further includes the following steps:

[0074] (1-2-1) Filter the collected point cloud data;

[0075] (1-2-2), pre-processing the filtered point cloud data;

[0076] (1-2-3) Update the temporary local map based on the preprocessed point cloud data.

[0077] The step (1-6) of comparing the first comparison error with the laser SLAM calibration error further includes the following steps:

[0078] (1-6-1) If the first comparison error is greater than the laser SLAM calibration error, the first IMU pose and motion information of the current frame is replaced with the lidar pose and motion information of the current frame;

[0079] (1-6-2) If the first comparison error is less than or equal to the laser SLAM calibration error, the first IMU pose and motion information of the current frame is stored.

[0080] The visual SLAM system in step (2) generates the second fused pose and motion information further comprising the following steps:

[0081] (2-1) Obtain the second IMU position and motion information of the current frame collected by the second IMU unit, and obtain the image pixel information of the current frame collected by the camera; in this embodiment, it is a binocular camera.

[0082] (2-2) performing front-end feature matching and tracking on the image pixel information of the current frame;

[0083] (2-3) triangulate the feature points, calculate the camera position and motion information of the current frame and the local map of the surrounding environment of the current frame in the visual system, and update the map in the visual system;

[0084] (2-4) combining the camera position and motion information of the previous frame with the camera position and motion information of the current frame to generate a third motion trajectory;

[0085] (2-5) combining the second IMU pose and motion information of the previous frame with the second IMU pose and motion information of the current frame to generate a fourth motion trajectory;

[0086] (2-6), comparing the third motion trajectory and the fourth motion trajectory to obtain a second comparison error, and comparing the second comparison error with a visual SLAM calibration error, wherein the visual SLAM calibration error is a system preset value;

[0087] (2-7) According to the comparison result of the second comparison error and the visual SLAM calibration error, the second IMU and camera posture and motion information are fused, and the second fused posture and motion information of the current frame in the visual system map is obtained in combination with the visual system map information.

[0088] Furthermore, in an embodiment of the present invention, the step (2-6) of comparing the second comparison error with the visual SLAM calibration error further includes the following steps:

[0089] (2-6-1) If the second comparison error is greater than the visual SLAM calibration error, the second IMU pose and motion information of the current frame is replaced with the camera pose and motion information of the current frame;

[0090] (2-6-2) If the second comparison error is less than or equal to the visual SLAM calibration error, store the second IMU pose and motion information of the current frame;

[0091] Furthermore, in an embodiment of the present invention, the step (3) of comparing the third comparison error with the system calibration error further includes the following steps:

[0092] (3-1), if the third comparison error is greater than the system calibration error, go to step (4), switch to the visual SLAM system, and combine the GPS reception signal to obtain the second fusion pose and motion information with the earth coordinate system;

[0093] (3-2) If the third comparison error is less than or equal to the system calibration error, the usage weights of the three-dimensional laser SLAM system and the visual SLAM system are calculated according to the weight algorithm. When there is an external GPS signal, the third fusion posture and motion information with the earth coordinate system is obtained in combination with the GPS receiving signal.

[0094] If the third comparison error in step (3-1) is greater than the system calibration error, the following steps are further included:

[0095] (3-1-1), comparing the third comparison error with the system switching error;

[0096] (3-1-2) If the third comparison error is less than or equal to the system switching error, the second fused pose and motion information is replaced with the first fused pose and motion information and stored in the visual SLAM system;

[0097] (3-1-3) If the third comparison error is greater than the system switching error, go to step (4) and switch to the visual SLAM system. When there is an external GPS signal, the second fused pose and motion information with the earth coordinate system is obtained by combining the GPS receiving signal.

[0098] See also Figure 7As shown, an embodiment of the present invention also provides a UAV system based on visual fusion laser radar SLAM positioning and navigation, including a three-dimensional laser SLAM system 100, a visual SLAM system 200, a flight control unit 300, and a system processing unit 400. The three-dimensional laser SLAM system 100 is used for UAV positioning and navigation. The three-dimensional laser SLAM system 100 provides first fused pose and motion information, the visual SLAM system 200 is used to optimize the three-dimensional laser SLAM system and provide UAV navigation when the three-dimensional laser SLAM system is not applicable. The visual SLAM system 200 provides second fused pose and motion information, the flight control unit 300 is used to control the UAV, and the system processing unit 400 is used to process data from each system. The system processing unit 400 compares the first fused pose and motion information with the second fused pose and motion information to obtain a third comparison error, and compares the third comparison error with a system calibration error preset by the system. When the third comparison error is greater than the system calibration error preset by the system, the current navigation system is switched to the visual SLAM system for UAV positioning and navigation. The system processing unit 400 includes a system calibration module 410, a system switching module 420 and a system fusion module 430.

[0099] The system calibration module 410 is used to correct the error of the visual SLAM system 200 in real time. The system calibration module compares the first fused pose and motion information with the second fused pose and motion information to obtain a third comparison error, and compares the third comparison error with a preset system calibration error;

[0100] The system switching module 420 is used for switching between the visual SLAM system 200 and the three-dimensional laser SLAM system 100;

[0101] The system fusion module 430 is used to fuse the posture and motion information of the visual SLAM system 200 and the three-dimensional laser SLAM system 100;

[0102] When the third comparison error is greater than the system calibration error, the third comparison error is compared with the preset system switching error; if the third comparison error is less than or equal to the system calibration error, the first fused pose and motion information and the second fused pose and motion information are output to the system fusion module;

[0103] If the third comparison error is less than or equal to the system switching error, the system calibration module replaces the second fused pose and motion information with the first fused pose and motion information, and outputs the first fused pose and motion information and the second fused pose and motion information to the system fusion module;

[0104] If the third comparison error is greater than the system switching error, the first fused pose and motion information and the second fused pose and motion information are output to the system switching module, and the system switching module only uses the second fused pose and motion information output by the visual SLAM system to locate and navigate the UAV;

[0105] The system fusion module calculates the usage weights of the 3D laser SLAM system and the visual SLAM system according to the results sent by the system calibration module and the weight algorithm, and fuses them to obtain the fused pose motion information.

[0106] Furthermore, in an embodiment of the present invention, a multi-sensor fusion module 440 is also included, which performs fusion based on the fused pose and motion information and updated global three-dimensional map information sent by the system fusion module. When there are external GPS and / or RTK signals, the GPS and / or RTK information are fused at the same time to obtain fused pose and motion information with the earth coordinate system.

[0107] Furthermore, in an embodiment of the present invention, the three-dimensional laser SLAM system 100 also includes a laser information acquisition unit 110 and a laser information processing unit 120, wherein the laser information acquisition unit 110 is used to collect information for the three-dimensional laser SLAM system 100, and the laser information processing unit 120 is used to process the information collected by the laser information acquisition unit.

[0108] The laser information acquisition unit 110 further includes a first IMU unit 111 and at least one laser radar 112, wherein:

[0109] The first IMU unit 111 is used to provide predicted pose and motion information for the three-dimensional laser SLAM system 100;

[0110] The laser radar 112 is used to collect point cloud data of the surrounding environment.

[0111] The laser radar 112 is a multi-line three-dimensional laser scanning radar.

[0112] The method of providing predicted posture and motion information for the three-dimensional laser SLAM system 100 also includes the first IMU unit 111 first collecting mileage information, converting the mileage information into drone posture change information through the drone inertial odometer kinematic model, and sending it to the Bayesian filter to preliminarily calculate the predicted posture and motion information.

[0113] Furthermore, in an embodiment of the present invention, the laser information processing unit 120 further includes a point cloud processing module 130, a radar data processing module 140, a radar system storage module 150, a radar system map update module 160 and a global map fusion module 170, wherein:

[0114] The point cloud processing module 130 is used to process point cloud data;

[0115] The radar data processing module 140 is used to process data from the radar system;

[0116] The radar system storage module 150 is used to store information generated by the laser information processing unit;

[0117] The radar system map update module 160 is used to update the map information of the three-dimensional laser SLAM system 100 in real time;

[0118] The global map fusion module 170 is used to fuse all local map information to generate global map information, so that the UAV can plan a navigation path in the global map information.

[0119] The point cloud processing module 130 further includes a filtering module 131 and a pre-processing module 132, wherein:

[0120] The filtering module 131 is used to filter the point cloud data collected by the laser radar 112;

[0121] The pre-processing module 132 is used to perform preliminary processing on the filtered point cloud data to obtain temporary local map information.

[0122] Furthermore, in an embodiment of the present invention, the filtering process includes denoising the point cloud data, removing abnormal points, and reducing the amount of redundant point cloud data.

[0123] The radar data processing module 140 further includes a radar solution module 141, a radar system trajectory generation module 142, a radar system trajectory comparison module 143, a radar system calibration module 144, and a lidar fusion module 145, wherein:

[0124] The radar solution module 141 is used to solve the filtered point cloud data to obtain the corresponding lidar pose and motion information and the local map information of the surrounding environment of the current frame;

[0125] The radar system trajectory generation module 142 is used to generate the motion trajectory of the UAV in two consecutive frames by combining the posture and motion information of two consecutive frames;

[0126] The radar system trajectory comparison module 143 is used to compare the motion trajectory and the error;

[0127] The radar system calibration module 144 is used to correct the error of the first IMU unit 111 in real time;

[0128] The lidar fusion module 145 is used to fuse the position and motion information output by the first IMU unit 111 and the lidar.

[0129] Furthermore, in an embodiment of the present invention, the visual SLAM system 200 also includes a visual information acquisition unit 210 and a visual information processing unit 220, wherein the visual information acquisition unit 210 is used to collect information for the visual SLAM system 200, and the visual information processing unit 220 is used to process the information collected by the visual information acquisition unit.

[0130] The visual information acquisition unit 210 further includes a second IMU unit 211 and at least one set of binocular cameras 212, wherein:

[0131] The second IMU unit 211 is used to provide predicted pose and motion information for the visual SLAM system 200;

[0132] The at least one set of binocular cameras 212 is used to capture image pixel information of the surrounding environment.

[0133] The provision of predicted pose and motion information for the visual SLAM system 200 also includes the second IMU unit 211 first collecting mileage information, converting the mileage information into UAV pose change information through the UAV inertial odometer kinematic model, and sending it to the Bayesian filter to preliminarily calculate the predicted pose and motion information.

[0134] The visual information processing unit 220 includes a feature processing module 230, a visual data processing module 240, a visual system storage module 250 and a visual system map updating module 260, wherein:

[0135] The feature processing module 230 is used to process image features;

[0136] The visual data processing module 240 is used to process data of the visual system;

[0137] The visual system storage module 250 is used to store information generated by the visual information processing unit;

[0138] The visual system map update module 260 is used to update the map information of the visual SLAM system 200 in real time.

[0139] The feature processing module 230 further includes a feature matching and tracking module 231 and a feature point processing module 232, wherein:

[0140] The feature matching and tracking module 231 is used to track the image pixel information collected by the binocular camera 212 in real time and match the image pixel information of two consecutive frames with features;

[0141] The feature point processing module 232 is used to process features into feature points.

[0142] The method of feature point processing is as follows:

[0143] A 6-dimensional vector multi-keyframe matching point triangulation feature point depth acquisition method is used to represent the 3D position of an environmental feature point, including the position of the drone's camera and the direction and depth information of the environmental feature points where the drone is located; the environment is continuously updated, and when the feature estimation covariance is less than a set threshold, the 6-dimensional representation of the environmental feature points where the drone is located is converted into 3-dimensional Euclidean coordinates. By establishing multiple feature points in the same frame image, the representation of the feature points belonging to the same frame is reduced to a camera posture plus multiple depths, so as to effectively reduce the length of the drone system state.

[0144] The visual data processing module 240 further includes a visual solution module 241, a visual system trajectory generation module 242, a visual system trajectory comparison module 243, a visual system calibration module 244, and a visual binocular fusion module 245, wherein:

[0145] The visual solution module 241 is used to solve the processed environmental feature point information into camera posture and motion information and local map information of the surrounding environment of the current frame;

[0146] The visual system trajectory generation module 242 is used to generate the motion trajectory of the UAV in two consecutive frames by combining the posture and motion information of two consecutive frames;

[0147] The visual system trajectory comparison module 243 is used to compare the motion trajectory and the error;

[0148] The visual system calibration module 244 is used to correct the error of the second IMU unit 211 in real time;

[0149] The visual binocular fusion module 245 is used to fuse the posture and motion information output by the second IMU unit 211 and the binocular camera 212.

[0150] The flight control unit 300 includes a global planning module 310, a local planning module 320 and a bottom control module 330, wherein:

[0151] The global planning module 310 is used to plan the optimal navigation path of the global map;

[0152] The local planning module 320 is used to plan the global optimal path based on the real-time local map information obtained after preprocessing;

[0153] The bottom control module 330 is used to control and distribute the UAVs.

[0154] The overall operation of a UAV system based on vision fusion 3D lidar SLAM positioning technology in this embodiment is as follows:

[0155] The 3D laser SLAM system 100 and the visual SLAM system 200 operate independently and generate first fused pose and motion information and second fused pose and motion information respectively, which are output to the system calibration module 410;

[0156] The system calibration module 410 compares the first fused pose and motion information with the second fused pose and motion information to obtain a third comparison error, and compares the third comparison error with a preset system calibration error;

[0157] If the third comparison error is greater than the system calibration error, the third comparison error is compared with a preset system switching error; if the third comparison error is less than or equal to the system calibration error, the first fused pose and motion information and the second fused pose and motion information are output to the system fusion module 430;

[0158] If the third comparison error is less than or equal to the system switching error, the system calibration module 410 replaces the second fused pose and motion information with the first fused pose and motion information, and outputs the first fused pose and motion information and the second fused pose and motion information to the system fusion module 430;

[0159] If the third comparison error is greater than the system switching error, the first fused pose and motion information and the second fused pose and motion information are output to the system switching module 420, and the system switching module 420 only uses the second fused pose and motion information output by the visual SLAM system 200 to locate the position of the UAV;

[0160] The multi-sensor fusion module 440 performs fusion based on the fused pose and motion information sent by the system fusion module and the updated global 3D map information. When there is an external GPS and / or RTK signal, the multi-sensor fusion module 440 also fuses the GPS and / or RTK information to obtain a third fused pose and motion information with an earth coordinate system.

[0161] The global planning module 310 plans the optimal navigation path for the UAV to reach the target point based on the user-specified location, map update information, and the third fused pose and motion information or the second fused pose and motion information, i.e., the shortest and obstacle-free navigation path, and outputs all data to the local planning module 320;

[0162] The local planning module 320 plans a local optimal navigation path based on the real-time local map information sent by the pre-processing module 132 in the laser information processing unit and the data output by the global planning module 310, compares it with the optimal navigation path to obtain the optimal path, and outputs the optimal path to the bottom control module 330;

[0163] The bottom control module 330 controls the drone according to the optimal path and controls the drone’s flight speed, angle, and direction.

[0164] The drone received the command and began to fly.

[0165] The three-dimensional laser SLAM system 100 operates as follows:

[0166] The first IMU unit 111 obtains the mileage information of the 3D laser SLAM system 100, and converts the mileage information into the UAV posture change information through the UAV inertial odometry kinematic model. The information is sent to the Bayesian filter to preliminarily calculate the first IMU posture and motion information of the current frame and output it to the radar system trajectory generation module 142. The multi-line 3D laser scanning radar collects the point cloud data information of the current frame and outputs it to the filtering module 131;

[0167] The filtering module 131 performs filtering processing on the collected point cloud data, such as noise reduction, removal of abnormal points, and reduction of redundant point cloud data volume, and outputs the filtered point cloud data to the radar solution module 141 and the pre-processing module 132 respectively;

[0168] The pre-processing module 132 performs preliminary processing on the filtered point cloud data to obtain temporary local map information and outputs it to the local planning module 320;

[0169] The radar solution module 141 solves the filtered point cloud data to obtain the lidar pose and motion information of the current frame and the local map information of the surrounding environment of the current frame in the laser system, and outputs the lidar pose and motion information of the current frame to the radar system trajectory generation module 142, and outputs the local map information of the surrounding environment of the current frame in the laser system to the laser system map update module 160;

[0170] The laser system map update module 160 updates the local map information of two consecutive frames in real time and outputs it to the global map fusion module 170;

[0171] The global map fusion module 170 fuses all local map information to generate global map information. When the UAV takes off from the same location next time, it can directly use the global map information and output the global map information to the lidar fusion module 145 and the global planning module 310.

[0172] The radar system trajectory generation module 142 combines the lidar pose and motion information of the previous frame with the lidar pose and motion information of the current frame to generate a first motion trajectory, combines the first IMU pose and motion information of the previous frame with the first IMU pose and motion information of the current frame to generate a second motion trajectory, and outputs the first motion trajectory and the second motion trajectory to the radar system trajectory comparison module 143;

[0173] The radar system trajectory comparison module 143 compares the first motion trajectory and the second motion trajectory to obtain a first comparison error, compares the first comparison error with a preset laser SLAM calibration error, and outputs the comparison result to the radar system calibration module 144;

[0174] If the first comparison error is greater than the laser SLAM calibration error, the radar system calibration module 144 replaces the first IMU pose and motion information of the current frame with the lidar pose and motion information of the current frame. If the first comparison error is less than or equal to the laser SLAM calibration error, the first IMU pose and motion information of the current frame is stored.

[0175] The posture and motion information of the first IMU and the lidar are output to the lidar fusion module 145. The lidar fusion module 145 fuses the posture and motion information of the first IMU and the lidar according to the fusion algorithm and combines it with the currently updated global map model to obtain the first fused posture and motion information of the current frame, and outputs the first fused posture and motion information of the current frame to the system calibration module 410.

[0176] The visual SLAM system 200 operates as follows:

[0177] The second IMU unit 211 obtains the mileage information of the visual SLAM system 200, and converts the mileage information into the UAV posture change information through the UAV inertial odometer kinematic model. The information is sent to the Bayesian filter to preliminarily calculate the second IMU posture and motion information of the current frame and output it to the visual system trajectory generation module 242. The binocular camera 212 collects the image pixels of the current frame and outputs them to the feature matching and tracking module 231;

[0178] The feature matching and tracking module 231 tracks the image pixel information collected by the binocular camera 212 in real time and matches the image pixel information of two consecutive frames and outputs the matched feature points to the feature point processing module 232;

[0179] The feature point processing module 232 performs feature point triangulation, specifically using a 6-dimensional vector multi-keyframe matching point triangulation feature point depth acquisition method to represent the three-dimensional position of an environmental feature point, including the position of the drone's camera and the direction and depth information of the environmental feature point in which the drone is located; continuously updating the environment, and when the feature estimation covariance is less than a certain set threshold, converting the 6-dimensional representation of the environmental feature point in which the drone is located into a 3-dimensional Euclidean coordinate, establishing multiple feature points in the same frame image, and reducing the representation of feature points belonging to the same frame to a form of a camera pose plus multiple depths, thereby effectively reducing the length of the drone system state. The processed environmental feature point information is output to the visual solution module 241;

[0180] The visual solution module 241 solves the processed environmental feature point information to obtain the camera pose and motion information of the current frame and the local map of the surrounding environment of the current frame in the visual system, and outputs the camera pose and motion information of the current frame to the visual system trajectory generation module 242, and outputs the local map information of the surrounding environment of the current frame in the visual system to the visual system map update module 260;

[0181] The visual system map update module 260 updates the map information of the visual SLAM system 200 in real time and outputs the map information to the visual binocular fusion module 245;

[0182] The visual system trajectory generation module 242 combines the camera pose and motion information of the previous frame with the camera pose and motion information of the current frame to generate a third motion trajectory, combines the second IMU pose and motion information of the previous frame with the second IMU pose and motion information of the current frame to generate a fourth motion trajectory, and outputs the third motion trajectory and the fourth motion trajectory to the visual system trajectory comparison module 243;

[0183] The visual system trajectory comparison module 243 compares the third motion trajectory and the fourth motion trajectory to obtain a second comparison error, compares the second comparison error with a preset visual SLAM calibration error, and outputs the comparison result to the visual system calibration module 244;

[0184] If the second comparison error is greater than the visual SLAM calibration error, the visual system calibration module 244 replaces the second IMU pose and motion information of the current frame with the camera pose and motion information of the current frame. If the second comparison error is less than or equal to the visual SLAM calibration error, the second IMU pose and motion information of the current frame is stored.

[0185] The posture and motion information of the second IMU and the binocular are output to the visual binocular fusion module 245. The visual binocular fusion module 245 fuses the posture and motion information of the second IMU and the camera according to the fusion algorithm, and combines the visual system map information to obtain the second fused posture and motion information of the current frame in the visual system map, and outputs the second fused posture and motion information of the current frame to the system calibration module 410.

[0186] The method and system provided by the present invention simultaneously use a visual system and a three-dimensional laser system based on a system fusion module. When a deviation occurs in the navigation positioning of one of the systems, it can seamlessly switch to the other system through the system switching module, thereby improving the environmental adaptability and stability of the UAV. At the same time, the system processing unit compares and corrects the data of the two systems in real time, making the system positioning and navigation more accurate and applicable to a wider range of scenarios.

[0187] At the same time, in order to avoid the influence of the error caused by the inertial odometry on the accuracy of UAV positioning and the effect of map construction, the present invention proposes a real-time calibration strategy of the inertial odometry. The laser SLAM system adopts the real-time calibration strategy of lidar + inertial odometry to realize the real-time correction of the odometry error, thereby improving the accuracy of UAV positioning and map construction of the SLAM algorithm.

[0188] In addition, compared with traditional methods, the feature point depth acquisition method of triangulation of multiple keyframe matching points does not require manual feature extraction or optical flow image extraction, does not require the construction of feature descriptors, does not require inter-frame feature matching, and does not require complex geometric calculations. It can enable new drones to continuously learn in depth and improve the overall reliability of the system.

[0189] Another important application scenario of the present invention is that when a drone using laser radar technology for navigation encounters heavy rain, thick smoke or dense fog, the data obtained by the laser radar will be inaccurate, resulting in the drone being unable to determine its own position and posture, which in turn makes it difficult for the drone to understand the complex environment and make motion decisions. To prevent the drone from being unable to make correct motion decisions in bad weather such as heavy rain, thick smoke, and dense fog, the drone provided by the embodiment of the present invention includes a visual SLAM system. When the controller detects that the error between the position and posture obtained by the laser odometer and the reference true value is greater than a certain set threshold, the laser SLAM system is switched to the visual SLAM system for navigation, and the visual SLAM system adopts a feature point depth acquisition method of triangulation of multiple keyframe matching points to effectively realize the navigation of the drone in adverse weather environments.

[0190] The above is a detailed introduction to the embodiments of a drone with an automatic switching navigation system and a visual fusion three-dimensional lidar SLAM positioning method and its application provided by the present invention. For those skilled in the art, based on the ideas of the embodiments of the present invention, there may be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A drone with an automatic switching navigation system, comprising a drone body, characterized in that Also includes: A laser SLAM system is used to achieve drone navigation. The laser SLAM system includes a laser radar unit and an IMU inertial unit. The laser radar unit and the IMU inertial unit are integrated into a frame. The laser radar unit is located in front of the drone body. The laser SLAM system uses the laser radar in combination with the inertial odometer in the IMU inertial unit to calibrate the odometer's position information in real time. A visual SLAM system is used to assist in the navigation of drones. The visual SLAM system includes a binocular camera unit located on the front of the drone body, and the binocular camera unit is located above the laser radar. The visual SLAM system uses a feature point depth acquisition method based on triangulation of multiple keyframe matching points to reconstruct the three-dimensional position of feature points. and a controller, when the controller detects that the error between the position and the reference true value obtained by the laser inertial odometer in the laser SLAM system is greater than the switching set threshold, the laser SLAM system is switched to the visual SLAM system for UAV navigation, including the following steps: (1) obtaining the first fused position and motion information generated by the three-dimensional laser SLAM system; (2) obtaining the second fused position and motion information generated by the visual SLAM system; (3) comparing the first fused position and motion information with the second fused position and motion information to obtain a third comparison error, and comparing the third comparison error with the system calibration error; (4) when the third comparison error ... and the laser SLAM system is switched to the visual SLAM system for UAV navigation, and the laser SLAM system is switched to the visual SLAM system for UAV navigation, and the laser SLAM system is switched to the visual SLAM system for UAV navigation, and the laser SLAM system is switched to the visual SLAM system for UAV navigation, and the laser SLAM system is switched to the visual SLAM system for UAV navigation, and the laser SLAM system is switched to the visual SLAM system for UAV navigation, and the laser SLAM system is switched to the visual SLAM system for UAV navigation, and the laser SLAM system is switched to the visual SLAM system for UAV navigation, and the laser SLAM system is switched to the visual SLAM system for UAV navigation, and the laser SL When the comparison error is greater than the system calibration error, the system is switched to the visual SLAM system for navigation of the UAV. The step (3) further includes (3-1) if the third comparison error is greater than the system calibration error, the system enters step (4), switches to the visual SLAM system, and obtains the second fused pose and motion information with the earth coordinate system in combination with the GPS receiving signal; (3-2) if the third comparison error is less than or equal to the system calibration error, the use weights of the three-dimensional laser SLAM system and the visual SLAM system are calculated according to the weight algorithm, and when there is an external GPS signal, the GPS receiving signal is combined to obtain the third fused pose and motion information with the earth coordinate system.

2. The drone with an automatic switching navigation system according to claim 1, characterized in that: The laser SLAM system uses a scan matching algorithm of laser radar point cloud data to calculate the posture of the UAV and use it as the reference true value of the inertial odometry posture. When the error between the posture obtained by the inertial odometry and the reference true value is greater than a first set threshold, a calibration operation is performed and the posture calculated after calibration is used to update the original posture of the inertial odometry. Otherwise, no calibration is performed.

3. The drone with an automatic switching navigation system according to claim 2, characterized in that: The method uses a scan matching algorithm based on lidar data to calculate the drone's pose and use it as a reference true value for the inertial odometry pose. The method includes continuously acquiring environmental scan information provided by the lidar, matching two frames of point cloud data, calculating the drone's displacement between two consecutive frames of laser point cloud data, and calculating the drone's real-time pose by calculating the drone's displacement between all laser frames and combining it with the drone's starting pose. The real-time pose serves as a reference true value for the inertial odometry pose.

4. The drone with an automatic switching navigation system according to claim 3, wherein: The method for acquiring the depth of feature points using triangulation of multiple keyframe matching points includes: using a six-dimensional vector multi-keyframe matching point triangulation feature point depth acquisition method to represent the three-dimensional position of an environmental feature point, including the position of the drone's binocular camera unit and the direction and depth information of the environmental feature points in which the drone is located; continuously updating the environment, and when the feature estimation covariance is less than a second set threshold, converting the six-dimensional representation of the environmental feature points in which the drone is located into three-dimensional Euclidean coordinates, and establishing multiple feature points in the same frame image, reducing the representation of the feature points belonging to the same frame to the form of a binocular camera unit posture plus multiple depths.

5. The drone with an automatic switching navigation system according to claim 4, characterized in that: The laser SLAM system also includes a laser system map update module and a global map fusion module. The laser system map update module is used to update the map information therein according to the drone posture information updated in real time by the laser SLAM system. The global map fusion module fuses the local map information of all local laser SLAM systems to generate a global three-dimensional map.

6. The drone with an automatic switching navigation system according to claim 5, characterized in that: The visual SLAM system also includes a visual system map update module, which updates the map information therein according to the drone posture information updated in real time by the visual SLAM system.

7. The UAV with an automatic switching navigation system according to claim 6, characterized in that: It also includes a multi-sensor fusion module, which fuses the UAV posture motion information obtained by the laser SLAM system, the UAV posture motion information obtained by the visual SLAM system, and the updated global three-dimensional map information to obtain optimized UAV posture motion information; when there are external GPS and / or RTK signals, the GPS and / or RTK information are simultaneously integrated to obtain the UAV posture motion information with the earth coordinate system.

Images