A SLAM system based on tightly coupled multi-source data

Abstract

The disclosed embodiment provides a SLAM system based on tightly coupled multi-source data, belonging to the field of data processing technology, and specifically includes: a laser radar subsystem, the laser radar subsystem is used to obtain point cloud data; a visual inertial subsystem, the visual inertial subsystem includes a visual camera and a depth camera; an inertial measurement unit, the inertial measurement unit user obtains inertial measurement values; a system control unit; a closed-loop detection module; a joint factor graph module, the laser radar subsystem, the visual inertial subsystem, and the inertial measurement unit are all connected to the closed-loop detection module, the closed-loop detection module is connected to the joint factor graph module, and the joint factor graph module is connected to the system control unit. Through the solution disclosed in the present invention, the efficiency, accuracy, and adaptability of mapping are improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of data processing technology, and in particular to a SLAM system based on tightly coupled multi-source data. Background Art

[0002] Simultaneous localization and mapping (SLAM) is a commonly used technology in computer vision. This technology uses sensors to acquire environmental data from the physical world and transmit it back to machines, enabling them to understand the unknown physical world. With the rapid development of technologies such as artificial intelligence, the internet, and computer hardware, the integration of physical and digital virtual environments, the internet and network technologies, and extended reality (also known as simultaneous localization and mapping) is driving new virtual interaction methods and technologies. SLAM provides the foundation for creating immersive interactive experiences for users.

[0003] The theory of single sensor methods is becoming increasingly sophisticated and has excellent performance on various data sets, but the complexity of actual application scenarios is often higher than that of specific data sets. For example, vision-based methods are suitable for location recognition and environments with complex textures, but changes in the lighting environment can cause performance losses. Therefore, other methods can usually be fused with vision-based methods to improve the robustness and accuracy of SLAM systems. In order to further improve system performance, the present invention combines multi-source data acquisition and fusion, and a tightly coupled approach to jointly optimize the overall system method.

[0004] Drones can carry out large-scale activities in indoor and outdoor spaces, and have the advantage of separating humans from machines. The positioning accuracy of traditional GPS robots is greatly affected by signals, and the SLAM algorithm will have scale uncertainty problems when the robot moves at high speed, affecting the closed-loop detection optimization performance.

[0005] It can be seen that there is an urgent need for a SLAM system based on tightly coupled multi-source data that can meet the mapping accuracy and efficiency requirements of scenes of different sizes, types, and lighting environments indoors and outdoors. Summary of the Invention

[0006] In view of this, the embodiments of the present disclosure provide a SLAM system based on tightly coupled multi-source data, which at least partially solves the problems of poor mapping efficiency, accuracy and adaptability in the prior art.

[0007] The present disclosure provides a SLAM system based on tightly coupled multi-source data, including:

[0008] LiDAR subsystem, the LiDAR subsystem is used to obtain point cloud data;

[0009] Visual-inertial subsystem, the visual-inertial subsystem includes a visual camera and a depth camera;

[0010] Inertial measurement unit, the inertial measurement unit is used to obtain inertial measurement values;

[0011] System control unit;

[0012] Closed-loop detection module;

[0013] The joint factor graph module, the lidar subsystem, the visual-inertial subsystem and the inertial measurement unit are all connected to the closed-loop detection module, the closed-loop detection module is connected to the joint factor graph module, and the joint factor graph module is connected to the system control unit. The system control unit is used to control the depth camera to obtain monocular images and depth information outdoors, and control the visual camera to obtain binocular image information indoors. The closed-loop detection module performs feature association, dynamic matching and loop detection on the monocular image, depth information, binocular image information and inertial measurement values, and finally completes positioning and mapping through global optimization of the joint factor graph.

[0014] According to a specific implementation of the embodiment of the present disclosure, the inertial measurement unit includes an inertial measurement pre-integration unit.

[0015] According to a specific implementation of the embodiment of the present disclosure, the visual-inertial subsystem further includes an alignment calculation unit and a frame detection unit.

[0016] According to a specific implementation of an embodiment of the present disclosure, the lidar subsystem further includes a feature point calculation unit and a scan matching unit.

[0017] According to a specific implementation of an embodiment of the present disclosure, the joint factor graph module includes a constraint updating unit.

[0018] According to a specific implementation of the embodiment of the present disclosure, the closed-loop detection module includes a candidate frame calculation unit.

[0019] According to a specific implementation method of an embodiment of the present disclosure, the system control unit is also used to control the initialization of the lidar subsystem and obtain depth information, system status and bias, align the laser frame and the visual frame through timestamps, and publish the pixel coordinates of all feature points tracked by the binocular at the same time.

[0020] According to a specific implementation of the embodiment of the present disclosure, the system control unit is further used to obtain the return data of the lidar subsystem, the visual inertial subsystem and the IMU module, take the initial value of the lidar subsystem and the inertial measurement value integration to complete system initialization, and the system state S can be expressed as:

[0021] S = [M, P, V, B]

[0022] Where M represents the rotation matrix, P represents the position vector, V represents the velocity, and B represents the deviation value.

[0023] According to a specific implementation of an embodiment of the present disclosure, the visual-inertial subsystem also includes a visual odometry, which is used to align the point cloud of the laser frame to obtain a sparse depth map, obtain the depth information of the feature to complete the visual inertial navigation initialization, combine the laser point cloud data to obtain a dense depth map, use a two-dimensional K-pile tree to find the connection between visual and lidar features, calculate the depth of the feature point by the length of the line connecting the visual camera center and the feature point, and the visual-inertial subsystem obtains an estimated value and completes initialization. The initial value of the scan matching is obtained through the system, that is, the pose output by the visual odometry is used as the initial value.

[0024] According to a specific implementation of an embodiment of the present disclosure, the closed-loop detection module includes visual odometry constraints, radar odometry constraints, inertial measurement constraints, and closed-loop detection constraints.

[0025] The SLAM system based on tightly coupled multi-source data in the embodiment of the present invention includes: a lidar subsystem, which is used to obtain point cloud data; a visual-inertial subsystem, which includes a visual camera and a depth camera; an inertial measurement unit, which is used to obtain inertial measurement values; a system control unit; a closed-loop detection module; a joint factor graph module, the lidar subsystem, the visual-inertial subsystem and the inertial measurement unit are all connected to the closed-loop detection module, the closed-loop detection module is connected to the joint factor graph module, and the joint factor graph module is connected to the system control unit. The system control unit is used to control the depth camera to obtain monocular images and depth information outdoors, and control the visual camera to obtain binocular image information indoors. After the closed-loop detection module performs feature association, dynamic matching and loop detection on the monocular image, depth information, binocular image information and inertial measurement values, it finally completes positioning and mapping through global optimization of the joint factor graph.

[0026] The beneficial effects of the embodiments of the present disclosure are as follows: through the scheme of the present disclosure, the lidar subsystem and the visual-inertial subsystem respectively obtain visual information and planar structure information, and the data are in the form of 3D point cloud data and 2D raster data respectively. The inertial measurement unit obtains inertial measurement values, and then the system control unit, closed-loop detection module and joint factor graph module preprocess and extract features of the obtained data and jointly optimize the data, finally completing positioning and mapping, thereby improving mapping efficiency, accuracy and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0028] Figure 1 A schematic structural diagram of a SLAM system based on tightly coupled multi-source data provided by an embodiment of the present disclosure;

[0029] Figure 2 A schematic diagram of a specific mounting framework of a SLAM system based on tightly coupled multi-source data provided by an embodiment of the present disclosure;

[0030] Figure 3 A schematic diagram of a three-dimensional point cloud map provided in an embodiment of the present disclosure;

[0031] Figure 4 A schematic diagram of the connection between laser characteristics provided by an embodiment of the present disclosure;

[0032] Figure 5 A schematic diagram of the data processing flow of a SLAM system based on tightly coupled multi-source data provided by an embodiment of the present disclosure.

[0033] Summary of reference numerals:

[0034] SLAM system 100 based on tightly coupled multi-source data;

[0035] LiDAR subsystem 110 , visual-inertial subsystem 120 , inertial measurement unit 130 , system control unit 140 , and joint factor graph module 150 . DETAILED DESCRIPTION

[0036] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0037] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0038] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0039] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0040] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0041] The embodiment of the present disclosure provides a SLAM system based on tightly coupled multi-source data, and the method can be applied to the process of environment acquisition and positioning.

[0042] See also Figure 1 , is a structural diagram of a SLAM system based on tightly coupled multi-source data provided by an embodiment of the present disclosure. Figure 1 As shown, the SLAM system 100 based on tightly coupled multi-source data mainly includes:

[0043] A laser radar subsystem 110 , which is used to acquire point cloud data;

[0044] The visual-inertial subsystem 120 includes a visual camera and a depth camera;

[0045] An inertial measurement unit 130 , which is used to obtain inertial measurement values;

[0046] System control unit 140;

[0047] Closed-loop detection module (not shown);

[0048] The joint factor graph module 150, the lidar subsystem 110, the visual-inertial subsystem 120, and the inertial measurement unit 130 are all connected to the closed-loop detection module, the closed-loop detection module is connected to the joint factor graph module 150, and the joint factor graph module 150 is connected to the system control unit 140. The system control unit 140 is used to control the depth camera to obtain monocular images and depth information outdoors, and control the visual camera to obtain binocular image information indoors. The closed-loop detection module performs feature association, dynamic matching, and loop closure detection on the monocular image, depth information, binocular image information, and inertial measurement values, and finally completes positioning and mapping through global optimization of the joint factor graph.

[0049] During specific assembly, the laser radar subsystem 110, the visual inertial subsystem 120 and the inertial measurement unit 130 can be mounted on an external device such as a drone. Figure 2 As shown, in order to collect images or distances at different positions, the lidar subsystem 110, the visual inertial subsystem 120 and the inertial measurement unit 130 are then connected to the closed-loop detection module, the closed-loop detection module is connected to the joint factor graph module 150, and the joint factor graph module 150 is connected to the system control unit 140.

[0050] During use, the system control unit 140 can control the depth camera to obtain monocular images and depth information outdoors, and control the visual camera to obtain binocular image information indoors. The closed-loop detection module performs feature association, dynamic matching, and loop detection on the monocular image, depth information, binocular image information, and inertial measurement values, and finally completes positioning and mapping through global optimization of the joint factor graph.

[0051] The SLAM system based on tightly coupled multi-source data provided in this embodiment obtains visual information and planar structure information through the lidar subsystem and the visual-inertial subsystem respectively. The data are in the form of 3D point cloud data and 2D raster data respectively. The inertial measurement unit obtains inertial measurement values, and then the system control unit, closed-loop detection module and joint factor graph module preprocess and extract features of the obtained data and jointly optimize the data to finally complete positioning and mapping, thereby improving mapping efficiency, accuracy and adaptability.

[0052] Furthermore, the inertial measurement unit 130 includes an inertial measurement pre-integration unit.

[0053] Optionally, the visual-inertial subsystem 120 further includes an alignment calculation unit and a frame detection unit.

[0054] Optionally, the lidar subsystem 110 further includes a feature point calculation unit and a scan matching unit.

[0055] Optionally, the joint factor graph module 150 includes a constraint updating unit.

[0056] Optionally, the closed-loop detection module includes a candidate frame calculation unit.

[0057] Furthermore, the system control unit 140 is also used to control the initialization of the lidar subsystem 110 and obtain depth information, system status and bias, align the laser frame and the visual frame through timestamps, and publish the pixel coordinates of all feature points tracked by the binocular at the same time.

[0058] Furthermore, the system control unit 140 is also used to obtain the return data of the lidar subsystem 110, the visual inertial subsystem 120 and the IMU module, and to take the initial value of the lidar subsystem 110 and the inertial measurement value integration to complete the system initialization. The system state S can be expressed as:

[0059] S = [M, P, V, B]

[0060] Where M represents the rotation matrix, P represents the position vector, V represents the velocity, and B represents the deviation value.

[0061] Furthermore, the visual-inertial subsystem 120 also includes a visual odometry, which is used to align the point cloud of the laser frame to obtain a sparse depth map, obtain the depth information of the feature to complete the visual inertial navigation initialization, combine the laser point cloud data to obtain a dense depth map, use a two-dimensional K-pile tree to find the connection between visual and lidar features, and calculate the depth of the feature point by the length of the line connecting the visual camera center and the feature point. The visual-inertial subsystem 120 obtains the estimated value and completes the initialization. The initial value of the scan matching is obtained through this system, that is, the pose output by the visual odometry is used as the initial value.

[0062] Furthermore, the closed-loop detection module includes visual odometry constraints, radar odometry constraints, inertial measurement constraints and closed-loop detection constraints.

[0063] The spatial laser vision fusion mapping process of the SLAM system 100 based on tightly coupled multi-source data will be described below with reference to a specific embodiment.

[0064] Step 1: The drone acquires data using its onboard RGB-D camera, binocular camera, and single-line LiDAR. The RGB-D camera is used to obtain monocular images and depth information outdoors, the binocular camera is used to obtain left and right eye images indoors, and the single-line LiDAR obtains information about the plane the drone is on.

[0065] Step 2: The IMU module obtains acceleration and angular velocity, integrates the IMU measurements, and obtains the drone's position information.

[0066] Step 3: Initialize the laser system to obtain depth information, system status, and bias, align the laser frame and visual frame through timestamps, and publish the pixel coordinates of all feature points tracked by the binocular at the same time.

[0067] Step 4: The system control unit 140 obtains the feedback data from the lidar subsystem 110, the vision subsystem and the IMU module, takes the initial value of the lidar subsystem 110 and integrates the IMU measurement value to complete system initialization.

[0068] Step 5: Figure 3 As shown in the figure, the visual odometry aligns the point cloud of the laser frame to obtain a sparse depth map to obtain the depth of the feature to complete the visual inertial navigation initialization, and combines the laser point cloud data to obtain a dense depth map.

[0069] Step 6: Figure 4 As shown, a two-dimensional K-pile tree (polar coordinates) is used to find the connection between visual and lidar features.

[0070] The length of the line connecting the two points is used to calculate the depth of the feature point. The visual system obtains the estimated value and completes the initialization. The initial value of the scan matching is obtained through the system, that is, the pose output by the visual system odometer is used as the initial value.

[0071] Step 7: Obtain the position and attitude information of the two radar keyframes as the initial value for scan matching. After the laser system is initialized in step 3, estimate the factor graph deviation, camera pose and velocity.

[0072] The visual system obtains the estimated value and completes the initialization. The initial value of the scan matching is obtained through the system, that is, the pose output by the visual system odometry is used as the initial value. The system state S can be expressed as:

[0073] S = [M, P, V, B]

[0074] Where M represents the rotation matrix, P represents the position vector, V represents the velocity, and B represents the deviation value.

[0075] Step 8: Use the joint factor graph to slide and complete the process optimization, where the joint factors include visual odometry constraints, radar odometry constraints, and loop closure constraints.

[0076] Step 9: Four main constraints for loop closure detection are added to the factor graph for optimization: the visual odometry constraint, the radar odometry constraint, the IMU constraint, and the loop closure detection constraint. This is then combined with the factor graph for global optimization to complete localization and mapping. The lidar odometry constraint is derived from scan matching, which matches the current lidar keyframe with the global feature map. Constraint candidate frames for loop closure detection are first provided by the vision system and then further optimized through scan matching.

[0077] like Figure 5 As shown in the figure, the system drives the image sensor, laser sensor, and IMU measurement unit to obtain relevant image, depth, IMU integral and other information. The system control module completes positioning and mapping through threads such as feature association, dynamic matching, and loop detection, and then globally optimizes the joint factor graph.

[0078] The units involved in the embodiments described in this disclosure may be implemented by software or hardware.

[0079] It should be understood that various parts of the present disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0080] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A SLAM system based on tightly coupled multi-source data, characterized in that: include: LiDAR subsystem, the LiDAR subsystem is used to obtain point cloud data; Visual-inertial subsystem, the visual-inertial subsystem includes a visual camera and a depth camera; Inertial measurement unit, the inertial measurement unit is used to obtain inertial measurement values; System control unit; Closed-loop detection module; The joint factor graph module, the lidar subsystem, the visual inertial subsystem and the inertial measurement unit are all connected to the closed-loop detection module, the closed-loop detection module is connected to the joint factor graph module, and the joint factor graph module is connected to the system control unit. The system control unit is used to control the depth camera to obtain monocular images and depth information outdoors, and control the visual camera to obtain binocular image information indoors. The closed-loop detection module performs feature association, dynamic matching and loop detection on the monocular image, depth information, binocular image information and inertial measurement values, and finally completes the positioning and mapping through global optimization of the joint factor graph. The system control unit is also used to control the initialization of the lidar subsystem and obtain depth information, system status and bias, align the laser frame and the visual frame through the timestamp, and publish the pixel coordinates of all feature points tracked by the binocular at the same time; The visual-inertial subsystem also includes a visual odometry, which is used to align the point cloud of the laser frame to obtain a sparse depth map, obtain the depth information of the features to complete the visual inertial navigation initialization, combine the laser point cloud data to obtain a dense depth map, use a two-dimensional K-pile tree to find the connection between visual and lidar features, and calculate the depth of the feature point by the length of the line connecting the visual camera center and the feature point. The visual-inertial subsystem obtains the estimated value and completes the initialization. The initial value of the scan matching is obtained through this system, that is, the pose output by the visual odometry is used as the initial value.

2. The system according to claim 1, characterized in that ,The inertial measurement unit includes an inertial measurement pre-integration unit.

3. The system according to claim 2, characterized in that ,The visual inertial subsystem also includes an alignment calculation unit and a frame ,detection unit.

4. The system according to claim 3, characterized in that ,The laser radar subsystem also includes a feature point calculation unit and a scan ,matching unit.

5. The system according to claim 4, characterized in that ,The joint factor graph module includes a constraint updating unit.

6. The system according to claim 5, characterized in that ,The closed-loop detection module includes a candidate frame calculation unit.

7. The system according to claim 6, characterized in that The system control unit is also used to obtain the return data of the lidar subsystem, the visual inertial subsystem and the IMU module, and to take the initial value of the lidar subsystem and the inertial measurement value integration to complete the system initialization. The system state S can be expressed as: ； Where M represents the rotation matrix, P represents the position vector, V represents the velocity, and B represents the deviation value.

8. The system according to claim 7, characterized in that ,The closed-loop detection module includes visual odometry constraints, radar odometry constraints, inertial measurement constraints and closed-loop detection constraints.

Images