Semantic SLAM method based on indoor dynamic scene

Through SegFormer semantic segmentation network and variable expansion core technology, dynamic feature points are eliminated in real time and camera pose estimation is used to use static feature points to solve the problem of inaccurate pose estimation in dynamic scenarios of visual SLAM system and improve the stability of the system Constructs dense point cloud maps with precision and accuracy.

CN119048926AInactive Publication Date: 2024-11-29GUILIN UNIV OF ELECTRONIC TECH +1
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411213913.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-31
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing visual SLAM systems are difficult to accurately estimate the camera position in dynamic scenarios, resulting in reduced positioning accuracy and robustness, especially due to error problems caused by the extraction of feature points of dynamic objects.

Method used

The SegFormer semantic segmentation network is used to combine RGB-D camera data to eliminate dynamic feature points in real time, process semantic segmentation results through variable expansion cores, use static feature points to perform camera pose estimation, and build a dense point cloud map.

Benefits of technology

It improves the accuracy and system stability of camera position estimation in dynamic scenarios, reduces errors, and enhances the real-time and positioning accuracy of the SLAM system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048926A_ABST
    Figure CN119048926A_ABST
Patent Text Reader

Abstract

The invention relates to a semantic SLAM (Simultaneous Localization and Mapping) method based on an indoor dynamic scene. The method comprises the following steps of: (1) loading a binary format word bag; (2) segmenting a dynamic object in a scene from an RGB image acquired by the RGB-D camera through a SegFormer semantic segmentation network; feature point extraction is carried out on the RGB image; (3) automatically adjusting an expansion kernel used by a semantic segmentation result according to different numbers of feature points extracted in the step (2); (4) removing dynamic feature points by combining the expanded mask information in the step (3); (5) performing camera pose estimation in a dynamic scene through the static feature points obtained in the step (4); and constructing a dense point cloud map according to the depth image. According to the method, a binary format word bag is used for performing system initialization, a SegFormer semantic segmentation network is used for performing semantic segmentation on a scene, dynamic objects in an environment can be effectively eliminated under the RGB-D condition, and a dense point cloud map with static features is constructed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an ORB-SLAM3 system and the field of semantic segmentation, and in particular to a semantic SLAM method based on indoor dynamic scenes. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) is a process that uses sensors such as cameras, lidar, and IMU to enable robots to build maps based on collected data in unknown environments and achieve self-positioning at the same time. This technology has been widely used in autonomous driving, drones, augmented reality and other fields. SLAM can be roughly divided into laser SLAM, visual SLAM, and multi-sensor fusion SLAM according to the different sensors used. Compared with laser SLAM, visual SLAM (Visual SALM, VSLAM) mainly uses cameras as sensors for data collection, such as monocular, binocular, RGB-D cameras, etc., to provide posture and position information for robots. At the same time, because of its low cost and rich information collection, it has become a research focus in this field. In 2020, Mur-Artal et al. proposed a more advanced visual SLAM system ORB-SLAM3 based on ORB-SLAM2. This algorithm is based on the static assumption of the scene. In actual applications, there will be many dynamic objects such as walking people, chairs that move with people, and moving cars. When more dynamic object feature points are extracted during the SLAM process, the camera pose estimation will produce serious errors, thereby reducing the positioning accuracy and robustness of the system, and even causing positioning and mapping failures.

[0003] Semantic SLAM combines neural networks to perform semantic segmentation on dynamic objects in the scene. The obtained semantic segmentation results distinguish static and dynamic objects in the scene, effectively removing dynamic object feature points and avoiding camera pose estimation errors caused by a large number of dynamic features during the SLAM process. However, semantic segmentation also faces the problems of inaccurate edge segmentation and poor real-time performance. Summary of the invention

[0004] The present invention is based on the ORB-SLAM3 system and improves upon the shortcomings of the prior art to provide a semantic SLAM method based on indoor dynamic scenes. This method can eliminate dynamic feature points in the environment that have a greater impact on the system accuracy in real time, use static feature points to estimate the camera pose and construct a dense point cloud map, improve the pose estimation accuracy in dynamic scenes, and improve the stability and real-time performance of the system.

[0005] The present invention is implemented by the following technical solution: a semantic SLAM method based on indoor dynamic scenes, specifically comprising the following steps:

[0006] (1) Load the binary format word bag;

[0007] (2) The RGB image captured by the RGB-D camera is segmented into dynamic objects in the scene through the SegFormer semantic segmentation network; at the same time, feature points are extracted from the RGB image;

[0008] (3) Automatically adjust the dilation kernel used in the semantic segmentation result according to the number of feature points extracted in step (2);

[0009] (4) Combine the mask information after expansion in step (3) to remove dynamic feature points;

[0010] (5) Using the static feature points obtained in step (4), the camera pose is estimated in a dynamic scene; at the same time, a dense point cloud map is constructed based on the depth image.

[0011] Compared with the traditional dynamic scene processing method, the system is more stable and more accurate after the variable expansion kernel is used to improve the semantic segmentation results;

[0012] In step (2), the RGB image captured by the RGB-D camera is segmented into dynamic objects in the scene through the SegFormer semantic segmentation network; at the same time, feature points are extracted from the RGB image, and the method is as follows:

[0013] For the RGB image obtained by the RGB-D camera, a semantic segmentation thread is started to process, which runs in parallel with the tracking thread, dense mapping thread, local mapping thread and loop detection thread. The SegFormer semantic segmentation network is used for semantic segmentation to obtain pixel-level semantic segmentation results. At the same time, feature point extraction of the RGB image is performed in the tracking thread.

[0014] In step (3), according to the different numbers of feature points extracted in step (2), the semantic segmentation results are expanded using different expansion kernels, and the method is as follows:

[0015] For the feature points and semantic segmentation results obtained in step (2), the number of extracted feature points is counted, and the coefficient of the expansion kernel is automatically adjusted according to the number of feature points. The more feature points there are, the larger the coefficient of the expansion kernel is, and vice versa. The variable expansion kernel is used to expand the semantic segmentation results;

[0016] At this point, the expansion processing of the semantic segmentation network segmentation results and the extraction of feature points are completed, and the processed semantic segmentation network segmentation results and the extracted feature points are saved separately;

[0017] In the step (4), the dynamic feature points are removed in combination with the mask information expanded in the step (3), and the method is as follows:

[0018] Using the dilation result obtained in step (3), the pixels located in the area with semantic labels "person" and "chair" are screened out and called dynamic areas;

[0019] Using the distribution of pixel positions in the dynamic area in the image, the feature points retained in step (3) are screened, and the feature points located in the dynamic area are removed, and finally the static feature points are obtained;

[0020] In the step (5), the camera pose is estimated in a dynamic scene using the static feature points obtained in the step (4); and a dense point cloud map is constructed based on the depth image, as follows:

[0021] After obtaining the static feature points, four threads are started synchronously: tracking thread, local mapping thread, loop detection thread and dense mapping thread. Among them, the tracking thread first completes the initialization of the camera pose and map points according to the static feature points, and then selects the reference key frame tracking. When the tracking is lost, the repositioning tracking is started, and the pose is further optimized through local map tracking. At the same time, it will determine whether the current frame needs to be created as a key frame according to the conditions; the key frame input in the local mapping thread comes from the newly created key frame in the tracking thread. In order to increase the number of local map points, the key frames in the local map will be re-featured to generate a new For map points, local BA will optimize the key frame poses and map points in the co-visual map at the same time, and delete inaccurate map points and redundant key frames after optimization; the loop detection thread queries the data set through binary format word bags to detect whether a closed loop is formed, calculates the Sim3 pose between the current key frame and the closed-loop candidate key frame, performs closed-loop fusion and essential graph optimization, and makes all key frame poses more accurate; the dense mapping thread generates a static feature point cloud in the camera coordinate system based on the static feature points and the depth image collected by the RGB-D camera, and then transforms the point cloud to the world coordinate system for point cloud stitching and point cloud optimization, and saves it as a global point cloud map;

[0022] The dense point cloud map construction process is as follows: first, the RGB image corresponding to the input key frame and the SegFormer semantic segmentation result are matched with the depth map data, and the point cloud in the camera coordinate system is obtained using the depth information and pixel position information. This point cloud is then transformed to the world coordinate system and saved as a global point cloud map. After global BA, the global point cloud map is adjusted with the updated pose. The mathematical formula for calculating the point cloud coordinates is:

[0023]

[0024] Where R and t are the rotation vector and translation vector of the camera respectively, K is the intrinsic parameter matrix of the camera, and s is the depth scale, which takes a fixed value. Assuming that the depth value d and the pixel position (u, v) are known, the spatial coordinates (x, y, z) of the point cloud can be calculated. The final conversion relationship between the spatial coordinates and the pixel coordinates is:

[0025]

[0026] This constructs a dense point cloud map.

[0027] Compared with the existing technology, this technical solution uses a bag-of-words file in binary format to speed up the system initialization speed, and uses SegFormer's outstanding efficiency in semantic segmentation to apply it in the SLAM system. It combines the ORB feature point matching method to eliminate dynamic feature points in the system environment, and uses more accurate static feature points for camera pose estimation and loop detection, and constructs a dense point cloud map.

[0028] This method can remove dynamic feature points that cause inaccurate pose estimation and erroneous loop detection in the ORB-SLAM3 system, while building a dense point cloud map to improve the efficiency and accuracy of the SLAM system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of a semantic SLAM method based on indoor dynamic scenes provided by the present invention.

[0030] Figure 2 It is a schematic diagram of the effect of using different expansion kernels to expand the mask in step (3) of a semantic SLAM method based on indoor dynamic scenes provided by the present invention.

[0031] Figure 3 It is the effect diagram of the embodiment. DETAILED DESCRIPTION

[0032] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] Example:

[0034] Reference Figure 1 , a semantic SLAM method based on indoor dynamic scenes, comprising the following steps:

[0035] Step 1: Load the binary format bag of words.

[0036] Step 2: Use the SegFormer semantic segmentation network to segment the RGB images captured by the RGB-D camera to identify dynamic objects in the scene; at the same time, extract feature points from the RGB images.

[0037] For the RGB image obtained by the RGB-D camera, a semantic segmentation thread is started to process, which runs in parallel with the tracking thread, dense mapping thread, local mapping thread and loop detection thread. The SegFormer semantic segmentation network is used for semantic segmentation to obtain pixel-level semantic segmentation results. At the same time, feature point extraction of the RGB image is performed in the tracking thread.

[0038] Step 3: According to the number of feature points extracted in step (2), the semantic segmentation results are expanded using different expansion kernels.

[0039] For the feature points and semantic segmentation results obtained in step (2), the number of extracted feature points is counted, and the coefficient of the expansion kernel is automatically adjusted according to the number of feature points. The more feature points there are, the larger the coefficient of the expansion kernel is, and vice versa. The variable expansion kernel is used to expand the semantic segmentation results. At this point, the expansion processing of the semantic segmentation network segmentation results and the extraction of feature points are completed, and the processed semantic segmentation network segmentation results and the extracted feature points are saved separately.

[0040] Step 4: Combine the mask information after expansion in step (3) to remove dynamic feature points.

[0041] Using the dilation result obtained in step (3), the pixels located in the areas with semantic labels "person" and "chair" are screened out, which are called dynamic areas. Using the distribution of pixel positions in the dynamic areas in the image, the feature points retained in step (3) are screened, and the feature points located in the dynamic areas are eliminated, and finally the static feature points are obtained.

[0042] Step 5: Use the static feature points obtained in step (4) to estimate the camera pose in a dynamic scene; at the same time, construct a dense point cloud map based on the depth image.

[0043] After obtaining the static feature points, four threads are started synchronously: tracking thread, local mapping thread, loop detection thread and dense mapping thread. Among them, the tracking thread first completes the initialization of the camera pose and map points according to the static feature points, and then selects the reference key frame tracking. When the tracking is lost, the repositioning tracking is started, and the pose is further optimized through local map tracking. At the same time, it will determine whether the current frame needs to be created as a key frame according to the conditions; the key frame input in the local mapping thread comes from the newly created key frame in the tracking thread. In order to increase the number of local map points, the key frames in the local map will be re-featured to generate a new For map points, local BA will optimize the key frame poses and map points in the co-visual map at the same time, and delete inaccurate map points and redundant key frames after optimization; the loop detection thread queries the data set through binary format word bags to detect whether a closed loop is formed, calculates the Sim3 pose between the current key frame and the closed-loop candidate key frame, performs closed-loop fusion and essential graph optimization, and makes all key frame poses more accurate; the dense mapping thread generates a static feature point cloud in the camera coordinate system based on the static feature points and the depth image collected by the RGB-D camera, and then transforms the point cloud to the world coordinate system for point cloud stitching and point cloud optimization, and saves it as a global point cloud map;

[0044] The dense point cloud map construction process is as follows: first, the RGB image corresponding to the input key frame and the SegFormer semantic segmentation result are matched with the depth map data, and the point cloud in the camera coordinate system is obtained using the depth information and pixel position information. This point cloud is then transformed to the world coordinate system and saved as a global point cloud map. After global BA, the global point cloud map is adjusted with the updated pose. The mathematical formula for calculating the point cloud coordinates is:

[0045]

[0046] Where R and t are the rotation vector and translation vector of the camera respectively, K is the intrinsic parameter matrix of the camera, and s is the depth scale, which takes a fixed value. Assuming that the depth value d and the pixel position (u, v) are known, the spatial coordinates (x, y, z) of the point cloud can be calculated. The final conversion relationship between the spatial coordinates and the pixel coordinates is:

[0047]

[0048] This constructs a dense point cloud map.

[0049] like Figure 1 As shown, this embodiment combines the ORB-SLAM3 system and the SegFormer semantic segmentation parallel operation method to eliminate dynamic feature points in the SLAM system in real time, reduce system errors, and obtain relatively stable static feature points; use stable static points for camera pose estimation and loop detection to construct more accurate global point cloud maps and dense point cloud maps.

[0050] This method is used to conduct experiments on an i5-13600KF CPU with 32GB of memory in Ubuntu 20.04, and then the TUM dataset is used to verify the system method of this example. Figure 3 Shown is the static feature dense point cloud map constructed by this example method.

Claims

1. A semantic SLAM method based on indoor dynamic scenes, characterized in that: The following steps are involved: (1) Load the binary format word bag; (2) The RGB image captured by the RGB-D camera is segmented into dynamic objects in the scene through the SegFormer semantic segmentation network; at the same time, feature points are extracted from the RGB image; (3) Automatically adjust the dilation kernel used in the semantic segmentation result according to the number of feature points extracted in step (2); (4) Combine the mask information after expansion in step (3) to remove dynamic feature points; (5) Using the static feature points obtained in step (4), the camera pose is estimated in a dynamic scene; at the same time, a dense point cloud map is constructed based on the depth image; The step (3) is specifically implemented as follows: For the feature points and semantic segmentation results obtained in step (2), the number of extracted feature points is counted, and the coefficient of the expansion kernel is automatically adjusted according to the number of feature points. The more feature points there are, the larger the coefficient of the expansion kernel is, and vice versa. The variable expansion kernel is used to expand the semantic segmentation results; At this point, the expansion processing of the semantic segmentation network segmentation results and the extraction of feature points are completed, and the processed semantic segmentation network segmentation results and the extracted feature points are saved separately; The step (4) is specifically implemented as follows: Using the dilation result obtained in step (3), the pixels located in the area with semantic labels "person" and "chair" are screened out and called dynamic areas; Using the distribution of pixel positions in the dynamic area in the image, the feature points retained in step (3) are screened, and the feature points located in the dynamic area are removed, and finally the static feature points are obtained; The step (5) is specifically implemented as follows: After obtaining the static feature points, four threads are started synchronously: tracking thread, local mapping thread, loop detection thread and dense mapping thread. Among them, the tracking thread first completes the initialization of the camera pose and map points according to the static feature points, and then selects the reference key frame tracking. When the tracking is lost, the repositioning tracking is started, and the pose is further optimized through local map tracking. At the same time, it will determine whether the current frame needs to be created as a key frame according to the conditions; the key frame input in the local mapping thread comes from the newly created key frame in the tracking thread. In order to increase the number of local map points, the key frames in the local map will be re-featured to generate a new For map points, local BA will optimize the key frame poses and map points in the co-visual map at the same time, and delete inaccurate map points and redundant key frames after optimization; the loop detection thread queries the data set through binary format word bags to detect whether a closed loop is formed, calculates the Sim3 pose between the current key frame and the closed-loop candidate key frame, performs closed-loop fusion and essential graph optimization, and makes all key frame poses more accurate; the dense mapping thread generates a static feature point cloud in the camera coordinate system based on the static feature points and the depth image collected by the RGB-D camera, and then transforms the point cloud to the world coordinate system for point cloud stitching and point cloud optimization, and saves it as a global point cloud map; The dense point cloud map construction process is as follows: first, the RGB image corresponding to the input key frame and the SegFormer semantic segmentation result are matched with the depth map data, and the point cloud in the camera coordinate system is obtained using the depth information and pixel position information. This point cloud is then transformed to the world coordinate system and saved as a global point cloud map. After global BA, the global point cloud map is adjusted with the updated pose. The mathematical formula for calculating the point cloud coordinates is: Where R and t are the rotation vector and translation vector of the camera respectively, K is the intrinsic parameter matrix of the camera, and s is the depth scale, which takes a fixed value. Assuming that the depth value d and the pixel position (u, v) are known, the spatial coordinates (x, y, z) of the point cloud can be calculated. The final conversion relationship between the spatial coordinates and the pixel coordinates is: This constructs a dense point cloud map.

2. The semantic SLAM method based on indoor dynamic scenes according to claim 1, characterized in that: In step (2), the RGB image captured by the RGB-D camera is segmented into dynamic objects in the scene through the SegFormer semantic segmentation network, and feature points are extracted from the RGB image at the same time, as follows: For the RGB image obtained by the RGB-D camera, a semantic segmentation thread is started to process, which runs in parallel with the tracking thread, dense mapping thread, local mapping thread and loop detection thread. The SegFormer semantic segmentation network is used for semantic segmentation to obtain pixel-level semantic segmentation results. At the same time, feature point extraction of the RGB image is performed in the tracking thread.

Citation Information

Cited By

  • SLAM method based on instance segmentation and dense mapping in dynamic environment

    CN119810248A

  • Visual SLAM method and system based on point-line fusion in dynamic scene

    CN119991806A

  • Method and system for carrying out spatial three-dimensional modeling by using mobile equipment

    CN120672968A