SLAM depth feature matching method for low-visibility mobile robot

Abstract

The invention provides a low-visibility environment-oriented mobile robot SLAM depth feature matching method, which is characterized by comprising the following steps of: 1, acquiring and storing original data of a laser radar, RGB-D and an IMU (inertial measurement unit) from a robot sensor, and preparing for preprocessing; step 2, matching the data acquired in the previous step, extracting time-synchronized data, and abandoning redundant data; 3, reading preset vehicle sensor installation position information, and correcting depth points of the RGB-D camera and the laser radar to be in the same coordinate system; 4, matching the depth points in the same coordinate system obtained in the previous step, retaining actually close point cloud pairs, and abandoning point cloud pairs which cannot be matched; and 5, constructing a complete SLAM system by using the steps and the ORB-SLAM2, and deploying the SLAM system on a mobile robot. According to the invention, mainly aiming at a low-visibility scene, a data acquisition and preprocessing method and a multi-sensor depth information fusion method are introduced into the SLAM system, so that the positioning and mapping effects in the low-visibility scene are more excellent.

Description

Technical Field

[0001] This invention relates to a deep feature matching method for SLAM of mobile robots in low visibility conditions. Background Technology

[0002] SLAM (Simultaneous Localization and Mapping) is a crucial technology for mobile robots and autonomous vehicles to achieve autonomous environmental perception and map building. In its early stages, SLAM relied on LiDAR as the primary perception sensor, acquiring the positions of feature points in the environment through laser scanning and reflection, and processing them into point clouds using algorithms such as NDT for localization and map building. Subsequently, with the development of computer vision technology, inexpensive RGB cameras, RGB-D cameras that provide additional depth information, and binocular cameras that mimic human binocular vision began to replace LiDAR as part of SLAM systems, and visual SLAM gradually gained a foothold.

[0003] However, both types of sensors have their limitations. While lidar can identify the location of a feature point in the vicinity and is unaffected by environmental factors, it cannot identify the object represented by that feature point. Visual sensors, on the other hand, can acquire information such as color and depth, but their operating principle leads to a decrease in the effective data they can obtain in low-visibility environments (such as dust or smog), making proper localization and mapping impossible. Summary of the Invention

[0004] To address the aforementioned technical shortcomings, this invention proposes a deep feature matching and fusion method for mobile robot SLAM systems in low-visibility environments. By combining LiDAR and RGB-D cameras, the SLAM system in this configuration can simultaneously utilize the environmentally unaffected nature of LiDAR and the rich semantic information of the camera.

[0005] This invention relates to a deep feature matching method for SLAM (Simultaneous Localization and Interaction) of mobile robots in low-visibility environments, characterized by comprising the following steps:

[0006] The first step is to collect raw data from the robot's sensors, including LiDAR, RGB-D, and IMU (Inertial Measurement Unit), and prepare it for preprocessing.

[0007] The second step is to match the data collected in the previous step, extract the time-synchronized data, and discard the redundant data.

[0008] The third step is to read the preset vehicle sensor installation location information and correct the depth points of the RGB-D camera and LiDAR to the same coordinate system.

[0009] Fourthly, the same coordinate system depth points obtained in the last step are matched, and the actual close point cloud pairs are reserved, and the point cloud pairs that cannot be matched are discarded;

[0010] Fifthly, the complete SLAM system is built by using the above steps and ORB-SLAM2, and is deployed on a mobile robot.

[0011] Beneficial effects: the application can effectively play the advantages of two sensors, and improve the mapping success rate and stability of the robot in a low-visibility environment.

[0012] The application effectively solves the problem that the positioning and mapping performance of the existing SLAM system is significantly reduced in a low-visibility environment. The traditional visual SLAM relies on image texture and lighting conditions. When there are obstructions such as dust, smoke, and haze in the environment, the number of effective pixels obtained by the RGB-D camera is greatly reduced, which leads to sparse feature points and significantly improves the matching failure rate, so that the system is difficult to complete stable pose estimation. However, the application cooperatively processes the laser radar point cloud and the RGB-D depth information, combines the advantages of high stability and high penetration of the laser radar in a low-visibility environment with the rich structure and semantic information of the RGB-D image, so that the SLAM system can still obtain continuous and reliable three-dimensional features in extremely weak visual conditions.

[0013] By proposing a depth feature matching fusion method, the application establishes a unified feature expression among multiple sensors, and generates more stable fusion depth features based on coordinate system conversion, point cloud projection, and depth completion technology, thereby significantly improving the success rate and robustness of feature matching. This method not only enhances the adaptability of SLAM in extreme scenes such as weak texture, low light, and high dust, but also effectively avoids the positioning failure problem of a single sensor in such environments.

[0014] Therefore, the method described in the application can maintain stable positioning and mapping capability in a low-visibility, non-GNSS available environment, and provides reliable protection for the autonomous navigation of mobile robots in complex environments such as industrial plants, underground spaces, and disaster areas, which has significant engineering application value and promotional significance. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 Flowchart of the technical route of the application

[0016] Figure 2 Distortion effect diagram of laser radar point cloud generated during movement

[0017] Figure 3 Carrier trolley used for system testing and running of the application

[0018] Figure 4 Positioning and mapping effect diagram of the system of the application on an unmanned vehicle DETAILED DESCRIPTION

[0019] In order to better understand the present application, the following will be combined with the accompanying Figures 1-4 to explain the embodiments of the present application in detail.

[0020] The present application is proposed due to the possibility of dust appearing in the oil mine practical training site existing in the school, the deficiency of the SLAM system of the single sensor for processing this scene, and the possibility of multi-sensor solving this difficulty. In the early stage of research, we analyzed the reasons for the insufficient feature point data collection of visual SLAM in low visibility and the implementation path of deep fusion by consulting existing SLAM related literature, so as to investigate the basic theory required by the research and the feasibility of implementing multi-sensor deep fusion positioning and navigation scheme under low visibility.

[0021] On the basis of the foregoing literature review, it is found that the implementation scheme has feasibility, and algorithm design is carried out. We find an ORB-SLAM2 as a basic framework, implement the formula in the basic theory by using code, and realize the lower data acquisition and preprocessing function, couple the above algorithm code with ORB-SLAM2, and build a complete SLAM system. In the test stage, although there are rich test data sets in the field of SLAM, there are few for low visibility environment, so we select a part of images and process the images in a simulated low visibility environment as the test data set of our project. The project is tested many times in simulation test to detect the integrity, effectiveness and stability of the function, find out the problems in time and modify, and optimize the system constantly, so that it reaches the best state.

[0022] Finally, real vehicle test. The SLAM system of the present application will be deployed on the test unmanned vehicle, and the real sensor is used to verify the sensor data time sequence synchronization effect and the quality of positioning and mapping under deep fusion, so as to judge the real effect of the system, which is used for summary and analysis.

[0023] As shown in Figure 1 The present application technology consists of three parts: data acquisition and preprocessing module, point cloud projection module and ORB-SLAM2 module. The data acquisition and preprocessing module communicates with ROS to obtain the original data on the sensor, synchronizes, generates matched data pairs, and uses IMU to correct the motion distortion of the laser radar to provide high-quality input for the point cloud projection module. In the point cloud projection module, the RGB-D camera and the laser radar are first converted in the coordinate system for depth point projection to avoid errors caused by installation position difference. Then the depth information provided by the camera image and the depth information provided by the laser radar point cloud will be fused according to the specified formula to generate a new single RGB-D frame, complete edge extraction, and enter the subsequent processing process of the ORB-SLAM2 module.

[0024] 1. Data acquisition and pre-processing module

[0025] The data acquisition and pre-processing module of the system is based on ROS, a famous open-source framework for software development of robotics applications. The vehicle can have a hardware signal synchronization mechanism, but the system assumes that this capability is not always present. In scenarios where the sensor hardware cannot achieve strict synchronization, the following software timing alignment and IMU-to-LiDAR correction procedures are used to ensure the spatio-temporal consistency and positioning accuracy of multi-sensor fusion.

[0026] ROS provides some standard named topics for data acquisition, including IMU (e.g. / imu / data, containing accelerometer and gyroscope raw measurements and their timestamps), RGB-D (e.g. / camera / color / image_raw and / camera / depth / image_raw or / camera / rgbd compound message), and LiDAR point cloud (e.g. / velodyne_points). These data headers all include timestamps, which are used as the synchronization judgment means of the system. For each type of sensor, the system sets up an independent ring buffer queue, with the cache element format of {timestamp t, raw message msg}. The cache length is set according to the maximum allowed delay and system load, generally covering 1-5 seconds of message volume to deal with peak delay and network jitter.

[0027] When using the data for processing, the closest-in-time other sensor sample is found for each "main reference" message in the cache, and if necessary, interpolation compensation is made for the pose / displacement. In this system, the main reference is the RGB-D camera. A maximum time tolerance threshold of 20-100 ms is set, or adjusted according to the actual situation of the hardware. If the time difference between the two sensors exceeds the threshold, it is determined that the frame cannot be reliably paired and is marked as mismatched or given a smaller weight in the back end. The ring buffer queue can allow long-staying data in the queue to be overwritten by new data, preventing the accumulation of excessive useless data.

[0028] The system can encounter situations where data other than the main reference does not meet the synchronization requirements and needs to be interpolated. For scalars such as translation vector p, linear interpolation can be used. Given the time coordinate pair and at the two time points found in the cache, the interpolation coefficient at the target time is described by equation 1.

[0029] (1)

[0030] ​For numerical values such as quaternions q, methods using spherical linear interpolation can be used.

[0031] With the previous data synchronization, the lidar data will be pre-integrated and corrected with the IMU data. Lidar is a periodic scanning sensor, and each scanning period takes some time, unlike cameras that collect information in an instant. If the lidar scans during the period of carrier motion, the point cloud will produce motion distortion, as shown in FIG. 2. The IMU can perceive angular velocity and acceleration at high frequency, and these velocity changes can be used to correct the distortion that occurs during motion. Equations 2 and 3 describe the method of correcting all radar data in each frame to the initial time. Figure 2

[0032] (2)

[0033] (3)

[0034] count and are the start and end timestamps of the radar frame, and the timestamp of any point is , is the pose transformation matrix, then the pose transformation matrix of this arbitrary point during motion can be described by . Then, all depth points are converted to the start time point .

[0035] 2. Point cloud projection module

[0036] In order to ensure that the system of the present application can still normally locate and map in a low-visibility environment, the system needs to use the depth information of the lidar to fuse into the RGB-D camera frame to compensate for the depth information lost due to insufficient visibility. In the present system, the RGB-D depth camera and the lidar are both mounted on a mobile carrier, but the installation positions and installation poses of the two types of sensors on the physical structure are not completely consistent, that is, the coordinate origins, coordinate axis directions, and field of view ranges of the two are different, as shown in FIG. 3. Therefore, the image coordinate system Figure 3 of the RGB-D sensor and the three-dimensional point cloud coordinate system of the lidar do not have a natural correspondence relationship. If the LiDAR point cloud and the RGB-D image are directly fused at the pixel level, problems such as coordinate misplacement, inconsistent depth, and feature point mismatch will occur, which cannot support the subsequent multi-sensor SLAM data association and feature matching.

[0037] Suppose the coordinates of a point in the lidar coordinate system are ​​​; camera coordinate system The coordinate of the midpoint Z is The coordinate in the image plane coordinate system is The extrinsic transformation matrix of the laser to the camera is And the intrinsic parameter of the camera is Formulas 4 and 5 describe the projection process from X to Y.

[0038] (4)

[0039] (5)

[0040] In formula 4, is the intrinsic matrix of the camera. In formula 5, by knowing the parameters Z, P, T and the existing laser radar coordinate system coordinate X, the to-be-solved coordinate Y in the plane coordinate system can be obtained.

[0041] 3. ORB-SLAM2 module

[0042] ORB-SLAM2 is a famous open-source visual SLAM system, and the system of the present application is obtained by modifying ORB-SLAM2. ORB-SLAM2 originally includes a front-end tracking module, a key frame module, a local mapping module, and a loop detection module. The two modules mentioned above are integrated as an RGB-D data preprocessing module, which is part of the front-end tracking of ORB-SLAM2.

[0043] 4. Construction of a complete SLAM system

[0044] Based on the above scheme description, the present application constructs a complete SLAM system and deploys it on an unmanned vehicle to test its positioning and mapping effect in a low-visibility scene. This SLAM system can complete the depth point information by adding new data acquisition and preprocessing modules and point cloud projection modules in a low-visibility environment through multiple sensors, and has achieved stronger positioning accuracy and mapping quality, as shown in FIG. 8. In the construction process of the present application, special attention is paid to the organic integration of the new modules and the original modules of ORB-SLAM2 to ensure that the new modules can correctly acquire and process data for subsequent process use. Figure 4

[0045] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.​

Claims

1. A low-visibility environment-oriented mobile robot SLAM depth feature matching method, characterized in that, It includes the following steps: S1, collect the raw data storage of laser radar, RGB-D and IMU (inertial measurement unit) from the robot sensor and prepare for preprocessing; S2, match the data collected in the last step, extract time-synchronized data, and discard redundant data; S3, read the preset vehicle sensor installation position information, and correct the depth points of the RGB-D camera and the laser radar to the same coordinate system; S4, match the depth points in the same coordinate system obtained in the last step, retain the actual close point cloud pairs, and discard the point cloud pairs that cannot be matched; S5, use the above steps and ORB-SLAM2 to build a complete SLAM system and deploy it on a mobile robot.

Images