Unmanned aerial vehicle autonomous positioning and mapping method and system based on visual inertia fusion

Through visual inertial fusion technology, the data of RGBD cameras and IMUs are used to realize high-precision positioning and mapping construction of drones in complex scenarios, and solve the problems of unstable positioning and poor mapping quality in complex environments by traditional methods. Provides efficient and robust positioning and mapping solutions.

CN118913259APending Publication Date: 2024-11-08HANGZHOU ZHUOXI INST OF BRAIN & INTELLIGENCE
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410968170.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional single sensor positioning methods have problems such as unstable positioning, poor mapping quality and poor robustness in complex scenarios or indoor environments, especially when GPS signals are unstable or unavailable.

Method used

Adopting the autonomous positioning and mapping method of drone based on visual inertial fusion, the depth image data and the inertial measurement unit IMU are obtained through the RGBD camera to obtain motion data, perform joint calibration of internal and external parameters, generate high-quality point cloud data, and match through point clouds. and optimize the autonomous positioning and mapping of drones.

Benefits of technology

It realizes real-time positioning and mapping with high precision and high robustness, and is suitable for UAV navigation in complex scenarios, improves the operating efficiency of the algorithm, and can be applied on lightweight devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118913259A_ABST
    Figure CN118913259A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an unmanned aerial vehicle autonomous positioning and mapping method and system based on visual inertia fusion, and the method comprises the steps: obtaining depth image data through an RGBD camera, obtaining unmanned aerial vehicle IMU data through an IMU, and carrying out the preprocessing of the IMU data, and obtaining IMU pre-integration data; performing internal and external parameter joint calibration on the depth image data and the IMU data to obtain an internal and external parameter matrix of the RGBD camera and the IMU; obtaining high-quality point cloud data based on the RGBD camera and the internal and external parameter matrix of the IMU, and constructing a local sub-map according to the IMU pre-integration data and the high-quality point cloud data; and matching and optimizing the current point cloud frame and the local sub-map to complete autonomous positioning and mapping of the unmanned aerial vehicle. The method utilizes the stability of depth data obtained by the RGBD camera and the low noise characteristic of the IMU, effectively overcomes the defects of a traditional visual method in a rapid motion and weak texture scene, has high precision and real-time performance, and is suitable for autonomous navigation and positioning application scenes of various unmanned aerial vehicles, robots and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a method and system for autonomous positioning and mapping of unmanned aerial vehicles based on visual-inertial fusion. Background Art

[0002] With the continuous expansion and development of drone applications, the demand for high-precision positioning and mapping of drones is increasing. However, the traditional single-sensor positioning method has some limitations, which has prompted researchers to explore more reliable and accurate positioning and mapping solutions. Especially in complex scenes or indoor environments, GPS signals may be unstable or unavailable, and pure visual SLAM may be limited by lighting conditions and texture loss, resulting in unstable positioning, poor mapping quality, poor robustness and other problems. Therefore, there is an urgent need for a method that can effectively achieve high-precision positioning and mapping of drones. Summary of the invention

[0003] The purpose of the present invention is to provide a method and system for autonomous positioning and mapping of unmanned aerial vehicles based on visual-inertial fusion, aiming to solve the above-mentioned problems in the prior art.

[0004] The embodiment of the present invention provides a method for autonomous positioning and mapping of a UAV based on visual-inertial fusion, comprising:

[0005] Acquire depth image data through an RGBD camera, acquire drone IMU data through an inertial measurement unit IMU, and pre-process the IMU data to obtain IMU pre-integration data;

[0006] Perform internal and external parameter joint calibration according to the depth image data and the IMU data to obtain the internal and external parameter matrices of the RGBD camera and the IMU;

[0007] Obtain high-quality point cloud data based on the RGBD camera and the internal and external parameter matrices of the IMU, and construct a local submap according to the IMU pre-integrated data and the high-quality point cloud data;

[0008] The current point cloud frame and the local sub-map are matched and optimized to complete the autonomous positioning and mapping of the UAV.

[0009] The embodiment of the present invention provides an autonomous positioning and mapping system for unmanned aerial vehicles based on visual inertial fusion, comprising:

[0010] A data module is used to obtain depth image data through an RGBD camera, obtain drone IMU data through an inertial measurement unit IMU, and pre-process the IMU data to obtain IMU pre-integration data;

[0011] A calibration module is used to perform internal and external parameter joint calibration according to the depth image data and the IMU data to obtain the internal and external parameter matrices of the RGBD camera and the IMU;

[0012] A construction module, used to obtain high-quality point cloud data based on the internal and external parameter matrices of the RGBD camera and the IMU, and to construct a local submap according to the IMU pre-integrated data and the high-quality point cloud data;

[0013] The positioning and mapping module is used to match and optimize the current point cloud frame and the local sub-map to complete the autonomous positioning and mapping of the drone.

[0014] An embodiment of the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the above-mentioned method for autonomous positioning and mapping of a drone based on visual-inertial fusion are implemented.

[0015] An embodiment of the present invention also provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by a processor, the steps of the above-mentioned method for autonomous positioning and mapping of unmanned aerial vehicles based on visual-inertial fusion are implemented.

[0016] The use of the embodiments of the present invention may include the following beneficial effects: the efficient point cloud stitching method proposed in the embodiments of the present invention improves the operating efficiency of the algorithm, so that it can be directly applied to lightweight equipment; in addition, the embodiments of the present invention fully utilize the respective advantages of the depth camera and the inertial measurement unit IMU, and propose a high-precision, highly robust real-time positioning and mapping method, which provides favorable conditions for the exploration and navigation of drones in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 It is a flow chart of a method for autonomous positioning and mapping of a UAV based on visual-inertial fusion according to an embodiment of the present invention;

[0019] Figure 2 is a flow chart of the RGBD visual-inertial fusion technology algorithm of an embodiment of the present invention;

[0020] Figure 3is a schematic diagram of the spatial position of sensors in an embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of a point cloud format according to an embodiment of the present invention;

[0022] Figure 5 It is a flowchart of the execution of the RGBD visual-inertial fusion technology system according to an embodiment of the present invention;

[0023] Figure 6 Schematic diagram of an autonomous positioning and mapping system for a UAV based on visual-inertial fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0025] Method Embodiment

[0026] According to an embodiment of the present invention, a method for autonomous positioning and mapping of a UAV based on visual-inertial fusion is provided. Figure 1 FIG. 1 is a flow chart of a method for autonomous positioning and mapping of a UAV based on visual-inertial fusion according to an embodiment of the present invention. Figure 1 As shown, the autonomous positioning and mapping method of a UAV based on visual inertial fusion according to an embodiment of the present invention specifically includes:

[0027] Step S101, acquiring depth image data through an RGBD camera, acquiring drone IMU data through an inertial measurement unit IMU, and preprocessing the IMU data to obtain IMU pre-integrated data;

[0028] Step S102, performing internal and external parameter joint calibration according to the depth image data and the IMU data to obtain the internal and external parameter matrices of the RGBD camera and the IMU, specifically includes:

[0029] The Zhang Zhengyou calibration method is used to obtain the intrinsic parameter matrix of the RGBD camera according to the depth image data, the imu-utils tool is used to obtain the intrinsic parameter matrix of the IMU according to the IMU data, the Kalibr tool is used to establish an objective function based on minimizing the reprojection error, the total linear acceleration error, and the total angular velocity error, and the nonlinear least squares method is used to solve the objective function to obtain the extrinsic parameter matrices of the RGBD camera and the IMU;

[0030] Step S103, obtaining high-quality point cloud data based on the internal and external parameter matrices of the RGBD camera and the IMU, and constructing a local submap according to the IMU pre-integrated data and the high-quality point cloud data, specifically includes:

[0031] The depth image data is enhanced by using a joint bilateral filtering method to obtain a high-quality depth image, the high-quality depth image is converted into two-dimensional point cloud data based on the internal and external parameter matrix of the RGBD camera, the two-dimensional point cloud data is stitched by using the external parameter matrix of the RGBD camera and the IMU, and the stitched point cloud data is used to generate high-quality point cloud data by using a straight-through filtering and radius filtering algorithm;

[0032] Step S104, matching and optimizing the current point cloud frame and the local sub-map to complete autonomous positioning and mapping of the drone, specifically includes:

[0033] The pose of the current point cloud frame is obtained by the ICP matching algorithm, and the pose change between the current point cloud frame and the adjacent point cloud frame is used as a constraint node. If the constraint node is less than or equal to the pose threshold, the current point cloud frame will not be added to the optimization queue;

[0034] If the constraint node is greater than the pose threshold, the current point cloud frame is added to the optimization queue and matched with the local sub-map. The matched local sub-map is optimized using loop detection. The global trajectory and map are obtained based on the optimized local sub-map to complete the autonomous positioning and mapping of the UAV.

[0035] The above technical solution of the embodiment of the present invention is described in detail below in combination with the specific situation of the UAV autonomous positioning and mapping method based on visual inertial fusion in the embodiment of the present invention.

[0036] The embodiment of the present invention proposes a method for autonomous positioning and mapping of a UAV based on RGBD visual-inertial fusion technology. The method firstly perceives the depth image information of the UAV's working space, combines the UAV's own motion state with the joint data of the inertial measurement unit (IMU) to realize the estimation of its own position and attitude in space; at the same time, the depth image data is converted into two-dimensional point cloud data, and a joint calibration matrix of internal and external parameters of multiple depth cameras and IMU is further established to realize point cloud stitching, and generate high-quality point cloud based on the point cloud filtering algorithm; finally, positioning and mapping are completed based on the Cartographer algorithm.

[0037] like Figure 2 As shown, an unmanned aerial vehicle positioning and mapping method based on RGBD visual inertial fusion technology proposed in an embodiment of the present invention includes the following steps:

[0038] S1, collects depth image data from RGBD camera and data from inertial measurement unit (IMU);

[0039] S2. Complete the joint calibration of internal and external parameters based on the depth image data and IMU data, including:

[0040] a. Based on Zhang Zhengyou's calibration method, complete the intrinsic calibration of the RGBD camera: Specifically, use the camera to shoot a checkerboard calibration plate of known size, and get the projection point m = [u, v] T , the corresponding 3D point is M = [X, Y, Z] T ,definition represents the augmented matrix of x, then the projection point With 3D points The relationship between can be described as:

[0041]

[0042] Where s is an arbitrary scale factor, R and t are external parameter matrices, and A represents the internal parameter matrix. The internal parameter matrix of each camera can be estimated by the least squares method.

[0043] b. Complete IMU sensor calibration: Specifically, collect data from the IMU in a stationary state. The probability distribution of the linear acceleration and angular acceleration errors of the IMU relative to the ground is:

[0044]

[0045] In the formula, σ a is η a The standard deviation, σ w is η w The standard deviation of .

[0046] The IMU follows a normal distribution during motion, and the offset relationship between time i and time i+1 is:

[0047]

[0048]

[0049] In the formula, is the linear acceleration offset at time i (m / s 2 ), angular acceleration offset at time i (rad / s 2 ); is the linear acceleration offset change from time i to time i+1 (m / s 2 ), is the change in angular acceleration offset from time i to time i+1 (rad / s 2 ), σ a,rw yes The standard deviation, σ w,rw yes The standard deviation of . By fitting the distribution function based on statistical theory, we finally get σ a , σ w , σ a,rw , σ w,rw The value of is the calibration result of IMU.

[0050] c. Complete the joint calibration of three RGBD cameras and IMU: fix the postures of the three RGBD cameras and IMU, such as Figure 3 As shown in the figure, the global reference system of the system is the IMU coordinate system. The timestamp translation between each camera and the IMU is estimated in turn. Then, the maximum likelihood estimation is used to establish the objective function based on minimizing the reprojection error, the total linear acceleration error, and the total angular velocity error. Finally, the nonlinear least squares method is used to solve the external parameter matrix. Specifically, the ROS-based throttle tool limits the frequency of each camera to 20Hz and the IMU frequency to 200Hz. The AprilGrid calibration version with a grid size of 0.0308m is made. The joint calibration of each camera and the IMU is completed based on the Kalibr tool. The calibration results are T1 = [R1, t1], T2 = [R2, t2], and T3 = [R3, t3], respectively. In the formula, R i ,t i (i=1,2,3) represents the rotation and translation matrices of each camera and IMU respectively.

[0051] S3, generating a two-dimensional point cloud based on the forward projection matrix and the camera internal and external parameters;

[0052] S4, generating high-quality point cloud based on point cloud filtering algorithm, including:

[0053] d. Enhance image depth information based on the joint bilateral filtering method and output high-quality depth map D i ;

[0054] e. Based on the camera internal and external parameter matrix, the depth map D i Convert to 2D point cloud P i , the specific diagram of the depth map and 2D point cloud is as follows Figure 4 As shown, compared to 3D point clouds, 2D point clouds do not include height values.

[0055] f. Using the external parameter matrix T of IMU and camera i , the point cloud P i Align with the world coordinate system to achieve point cloud stitching. The final point cloud P is given by the following formula:

[0056]

[0057] g. Based on the radius filtering algorithm, count the number of point clouds within the radius, determine the confidence threshold, and generate high-quality point clouds

[0058] S5. Preprocess the IMU data (remove noise, etc., to enhance the robustness of the algorithm), obtain IMU pre-integrated data, and establish a sub-map based on the IMU pre-integrated data and point cloud data, specifically including:

[0059] h. Generate the corresponding velocity, displacement, and angular velocity increments based on IMU pre-integration to estimate the initial pose of the next visual SLAM front-end matching. The angle change trusts the IMU data, and the displacement change trusts the visual SLAM back-end output data, and these two are used as the initial values ​​of scan matching;

[0060] i. Construct the optimal sub-map based on adaptive voxel filtering and Ceres optimization method.

[0061] S6. Complete the matching of the current point cloud frame and the sub-map to achieve global map optimization.

[0062] The embodiment of the present invention chooses to integrate the information of the RGBD camera and the inertial measurement unit (IMU), wherein the RGBD camera can provide rich three-dimensional spatial information, including color images and depth images, so that the drone can better perceive the environment; and the IMU can provide high-precision attitude measurement, including angular velocity and acceleration information, which can reflect the motion state of the drone in real time. By complementing these two types of information, the real-time estimation of the position and attitude of the drone in space can be realized, thereby achieving high-precision positioning and mapping effects.

[0063] The embodiment of the present invention provides a method for autonomous positioning and mapping of unmanned aerial vehicles based on RGBD visual-inertial fusion technology, which uses RGBD camera and IMU data information for fusion to achieve high-precision autonomous positioning and scene mapping. Specifically, the method uses an RGBD camera to obtain event depth data streams in the environment in real time, and converts the depth data into high-quality point cloud data; the acceleration and angular velocity of the unmanned aerial vehicle are measured based on the IMU, and the position change of the unmanned aerial vehicle is obtained using a kinematic model; based on the data provided by the IMU and RGBD, a point cloud filtering algorithm is combined to generate a high-quality point cloud, and finally a scene grid map is constructed based on the Cartographer algorithm, thereby improving the accuracy of unmanned aerial vehicle positioning and mapping.

[0064] In the embodiment of the present invention, the advantage of using an RGBD camera is that it can accurately obtain depth data streams in complex scenes while having lower power consumption and latency. By fusing with IMU data, the respective advantages of the RGBD camera and IMU can be fully utilized while making up for their deficiencies. Through visual inertial fusion technology, high-precision autonomous positioning and mapping can be achieved, which has high real-time performance and accuracy and is suitable for application scenarios of autonomous navigation and positioning of drones.

[0065] In the embodiments of the present invention, a series of mathematical methods and algorithms are used, including but not limited to:

[0066] (1) Calibration of internal and external parameters of RGBD camera and IMU;

[0067] (2) Depth data is converted into point cloud data, point cloud filtering, etc.;

[0068] (3) Preprocessing of IMU data and construction of kinematic models, including angular velocity integration, acceleration filtering, and noise compensation;

[0069] (4) Positioning and mapping based on RGBD and IMU data, including scan matching and loop closure detection.

[0070] The overall process of the embodiment of the present invention is as follows Figure 5 As shown, first, it is necessary to complete the joint calibration of the internal and external parameters of the RGBD camera and the IMU. The camera internal parameter calibration adopts the Zhang Zhengyou calibration algorithm. Before performing the external parameter calibration, ensure that the camera sensors are fixed on the UAV platform and the calibration plate is fixed in front of the UAV. The embodiment of the present invention adopts the Kalibr tool to complete the camera and IMU external parameter calibration, and the imu-utils tool is used to complete the IMU internal parameter calibration.

[0071] The depth image data is converted into two-dimensional point cloud data. Specifically, based on the above-mentioned internal and external calibration data, the depth data generated by each camera is converted into a two-dimensional point cloud format and converted into the world coordinate system. At the same time, the straight-through filtering and radius filtering algorithms are used to remove abnormal point clouds and retain high-quality point clouds to input into the Cartographer algorithm. The principle of the radius filtering algorithm is as follows:

[0072] 1. For each point cloud P in space i (x i ,y i ,z i ), find all the neighboring points N(P i );

[0073] 2. Calculate P i The distance to its neighboring points is determined by setting a threshold R: if the farthest point among all neighboring points is iThe distance is less than the threshold R, then point P i If a point is considered as a core point, it is retained; otherwise, it is considered as an outlier and is removed.

[0074] The IMU data is pre-integrated to improve its data accuracy and reduce the integral cumulative error. At the same time, the timestamps of the IMU and point cloud data are calculated based on the linear interpolation algorithm.

[0075] Construct globally consistent pose and map: Specifically, the pose of the current point cloud frame is solved based on the ICP matching algorithm, the pose change ΔP between the previous and next two frames is used as the constraint node, and the pose threshold P is set. t The initial value of P; further construct a local sub-graph based on the adjacent multi-frame point cloud for further point cloud registration and loop detection; when ΔP≤P t , the frame point cloud is not added to the optimization queue, otherwise, the frame point cloud is added to the optimization queue and is ready to complete the matching with the local sub-map; finally, based on the above matching and optimization results, a globally consistent trajectory and map are obtained.

[0076] It should be noted that the sub-graph is represented in the form of an occupied grid map. The size of each grid is determined by the resolution set by the algorithm. Each grid has two states, representing the probability of being occupied and the probability of being blank. The two values ​​of the unknown grid are both 0.5. When the grid occupancy rate is lower than p min When the grid occupancy rate is higher than p max When the grid occupancy rate is in the interval [p min ,p max ], the grid status is considered unknown.

[0077] Based on p hit =1 and p miss = 0 Update the probability value of the grid, M new(x) as follows:

[0078]

[0079] In the formula, clamp is an interval-limited function. When the value in the brackets is greater than max, the maximum probability value p is returned. max When the value in the brackets is less than min, the minimum probability p is returned. min , otherwise return the current value, M old Represents the original probability value in the raster.

[0080] The map is optimized based on loop detection. If a loop relationship is detected under the current posture, relevant constraints are established. This method can correct the accumulated errors generated in the process of global map construction and ensure the accuracy of posture estimation.

[0081] In summary, the embodiment of the present invention proposes an autonomous positioning and mapping method based on RGBD visual inertial fusion technology. The method acquires scene point cloud based on three RGBD cameras, fuses IMU data, and completes fully autonomous scene mapping. The method is suitable for dark and complex scenes such as underground pipe galleries, and has the following beneficial effects:

[0082] 1. Based on three RGBD cameras, the scene depth image is obtained and the depth data is converted into two-dimensional point cloud data of the scene, which maximizes the scene information of the UAV workspace and realizes the map construction of complex scenes.

[0083] 2. The fusion of IMU data and RGBD data greatly reduces the positioning error during the UAV movement process and improves the positioning accuracy and robustness of the system.

[0084] 3. By constructing a grid map of the scene where the drone is located, it can provide a basis for the subsequent relocation and path planning of the drone.

[0085] 4. The efficient point cloud stitching solution improves the operating efficiency of the algorithm, allowing it to be directly applied to lightweight equipment.

[0086] System Example

[0087] According to an embodiment of the present invention, a UAV autonomous positioning and mapping system based on visual inertial fusion is provided. Figure 6 FIG. 1 is a schematic diagram of a UAV autonomous positioning and mapping system based on visual inertial fusion according to an embodiment of the present invention. Figure 6 As shown, the autonomous positioning and mapping system for unmanned aerial vehicles based on visual inertial fusion according to an embodiment of the present invention specifically includes:

[0088] The data module 60 is used to obtain depth image data through the RGBD camera, obtain drone IMU data through the inertial measurement unit IMU, and pre-process the IMU data to obtain IMU pre-integrated data;

[0089] The calibration module 62 is used to perform internal and external parameter joint calibration according to the depth image data and the IMU data to obtain the internal and external parameter matrices of the RGBD camera and the IMU, specifically for:

[0090] The Zhang Zhengyou calibration method is used to obtain the intrinsic parameter matrix of the RGBD camera according to the depth image data, the imu-utils tool is used to obtain the intrinsic parameter matrix of the IMU according to the IMU data, the Kalibr tool is used to establish an objective function based on minimizing the reprojection error, the total linear acceleration error, and the total angular velocity error, and the nonlinear least squares method is used to solve the objective function to obtain the extrinsic parameter matrices of the RGBD camera and the IMU;

[0091] The construction module 64 is used to obtain high-quality point cloud data based on the internal and external parameter matrices of the RGBD camera and the IMU, and to construct a local sub-map according to the IMU pre-integrated data and the high-quality point cloud data, specifically for:

[0092] The depth image data is enhanced by using a joint bilateral filtering method to obtain a high-quality depth image, the high-quality depth image is converted into two-dimensional point cloud data based on the internal and external parameter matrix of the RGBD camera, the two-dimensional point cloud data is stitched by using the external parameter matrix of the RGBD camera and the IMU, and the stitched point cloud data is used to generate high-quality point cloud data by using a straight-through filtering and radius filtering algorithm;

[0093] The positioning and mapping module 66 is used to match and optimize the current point cloud frame and the local sub-map to complete the autonomous positioning and mapping of the drone, and is specifically used for:

[0094] The pose of the current point cloud frame is obtained by the ICP matching algorithm, and the pose change between the current point cloud frame and the adjacent point cloud frame is used as a constraint node. If the constraint node is less than or equal to the pose threshold, the current point cloud frame will not be added to the optimization queue;

[0095] If the constraint node is greater than the pose threshold, the current point cloud frame is added to the optimization queue and matched with the local sub-map. The matched local sub-map is optimized using loop detection. The global trajectory and map are obtained based on the optimized local sub-map to complete the autonomous positioning and mapping of the UAV.

[0096] The embodiment of the present invention is a system embodiment corresponding to the above-mentioned method embodiment. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0097] In summary, the autonomous positioning and mapping method based on visual-inertial fusion technology proposed in the embodiment of the present invention has many advantages. First, it can achieve high-precision and high-real-time positioning and mapping, which is suitable for fast movement, complex environments and indoor scenes; second, this method can be implemented on different hardware platforms, making its application range wider; in addition, visual-inertial fusion technology can also adapt to different motion modes, including rotation, translation and hovering, etc., which provides strong support for autonomous navigation and environmental detection of drones. Therefore, the drone positioning and mapping method based on visual-inertial fusion technology has broad application prospects in drone applications. It can provide high-quality solutions for drones' intelligent cruising, precise navigation, environmental perception and obstacle avoidance tasks, and bring more possibilities for the further development and application of drone technology. The beneficial effects of the embodiments of the present invention specifically include:

[0098] (1) Through the data fusion of RGBD camera and IMU, high-precision autonomous positioning and mapping can be achieved, with high real-time performance and accuracy;

[0099] (2) Through visual-inertial fusion technology, the advantages of RGBD cameras and IMUs are fully utilized while making up for their shortcomings;

[0100] (3) Combining multiple RGBD camera data, it is possible to fully collect information about the drone’s surrounding environment and achieve high-precision map construction;

[0101] (4) Through point cloud filtering and efficient point cloud stitching methods, the operating efficiency of the algorithm is improved, so that the algorithm can be directly applied to lightweight equipment.

[0102] Device Example 1

[0103] An embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps described in the method embodiment when executed by the processor.

[0104] Device Example 2

[0105] An embodiment of the present invention provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by a processor, the steps described in the method embodiment are implemented.

[0106] The computer-readable storage medium in this embodiment includes, but is not limited to, ROM, RAM, magnetic disk or optical disk, etc.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for autonomous positioning and mapping of unmanned aerial vehicles based on visual-inertial fusion, characterized in that: include: Acquire depth image data through an RGBD camera, acquire drone IMU data through an inertial measurement unit IMU, and pre-process the IMU data to obtain IMU pre-integration data; Perform internal and external parameter joint calibration according to the depth image data and the IMU data to obtain the internal and external parameter matrices of the RGBD camera and the IMU; Obtain high-quality point cloud data based on the RGBD camera and the internal and external parameter matrices of the IMU, and construct a local submap according to the IMU pre-integrated data and the high-quality point cloud data; The current point cloud frame and the local sub-map are matched and optimized to complete the autonomous positioning and mapping of the UAV.

2. The method according to claim 1, characterized in that The internal and external parameters are jointly calibrated according to the depth image data and the IMU data to obtain the internal and external parameter matrices of the RGBD camera and the IMU, specifically including: The Zhang Zhengyou calibration method is used to obtain the intrinsic parameter matrix of the RGBD camera according to the depth image data. The imu-utils tool is used to obtain the intrinsic parameter matrix of the IMU according to the IMU data. The Kalibr tool is used to establish an objective function based on minimizing the reprojection error, the total linear acceleration error, and the total angular velocity error. The nonlinear least squares method is used to solve the objective function to obtain the extrinsic parameter matrices of the RGBD camera and the IMU.

3. The method according to claim 1, characterized in that The high-quality point cloud data obtained based on the internal and external parameter matrices of the RGBD camera and the IMU specifically includes: The depth image data is enhanced by using a joint bilateral filtering method to obtain a high-quality depth image. The high-quality depth image is converted into two-dimensional point cloud data based on the internal and external parameter matrices of the RGBD camera. The two-dimensional point cloud data is stitched using the external parameter matrices of the RGBD camera and the IMU. The stitched point cloud data is used to generate high-quality point cloud data using a straight-through filtering and radius filtering algorithm.

4. The method according to claim 1, characterized in that: Matching and optimizing the current point cloud frame and the local sub-map to complete the autonomous positioning and mapping of the drone specifically includes: The pose of the current point cloud frame is obtained by the ICP matching algorithm, and the pose change between the current point cloud frame and the adjacent point cloud frame is used as a constraint node. If the constraint node is less than or equal to the pose threshold, the current point cloud frame will not be added to the optimization queue; If the constraint node is greater than the pose threshold, the current point cloud frame is added to the optimization queue and matched with the local sub-map. The matched local sub-map is optimized using loop detection. The global trajectory and map are obtained based on the optimized local sub-map to complete the autonomous positioning and mapping of the UAV.

5. An autonomous positioning and mapping system for unmanned aerial vehicles based on visual inertial fusion, characterized in that: include: A data module is used to obtain depth image data through an RGBD camera, obtain drone IMU data through an inertial measurement unit IMU, and pre-process the IMU data to obtain IMU pre-integration data; A calibration module is used to perform internal and external parameter joint calibration according to the depth image data and the IMU data to obtain the internal and external parameter matrices of the RGBD camera and the IMU; A construction module, used to obtain high-quality point cloud data based on the internal and external parameter matrices of the RGBD camera and the IMU, and to construct a local submap according to the IMU pre-integrated data and the high-quality point cloud data; The positioning and mapping module is used to match and optimize the current point cloud frame and the local sub-map to complete the autonomous positioning and mapping of the drone.

6. The system according to claim 5, characterized in that The calibration module is specifically used for: The Zhang Zhengyou calibration method is used to obtain the intrinsic parameter matrix of the RGBD camera according to the depth image data. The imu-utils tool is used to obtain the intrinsic parameter matrix of the IMU according to the IMU data. The Kalibr tool is used to establish an objective function based on minimizing the reprojection error, the total linear acceleration error, and the total angular velocity error. The nonlinear least squares method is used to solve the objective function to obtain the extrinsic parameter matrices of the RGBD camera and the IMU.

7. The system according to claim 5, characterized in that The building blocks are specifically used for: The depth image data is enhanced by using a joint bilateral filtering method to obtain a high-quality depth image. The high-quality depth image is converted into two-dimensional point cloud data based on the internal and external parameter matrices of the RGBD camera. The two-dimensional point cloud data is stitched using the external parameter matrices of the RGBD camera and the IMU. The stitched point cloud data is used to generate high-quality point cloud data using a straight-through filtering and radius filtering algorithm.

8. The system according to claim 5, characterized in that The positioning and mapping module is specifically used for: The pose of the current point cloud frame is obtained by the ICP matching algorithm, and the pose change between the current point cloud frame and the adjacent point cloud frame is used as a constraint node. If the constraint node is less than or equal to the pose threshold, the current point cloud frame will not be added to the optimization queue; If the constraint node is greater than the pose threshold, the current point cloud frame is added to the optimization queue and matched with the local sub-map. The matched local sub-map is optimized using loop detection. The global trajectory and map are obtained based on the optimized local sub-map to complete the autonomous positioning and mapping of the UAV.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the method for autonomous positioning and mapping of a drone based on visual-inertial fusion as described in any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by the processor, the steps of the autonomous positioning and mapping method for unmanned aerial vehicle based on visual-inertial fusion as described in any one of claims 1 to 4 are implemented.

Citation Information

Cited By

  • Automatic calibration method and system for external parameters of multiple sensors of mowing robot

    CN120997301A