Method for enhancing descriptor semantic vision SLAM (Simultaneous Localization and Mapping) in dynamic environment

Abstract

The invention relates to the technical field of visual synchronous localization and mapping, in particular to a method for enhancing descriptor semantic vision SLAM (Simultaneous Localization and Mapping) in a dynamic environment, which comprises the following steps of: 1, performing semantic segmentation on a scene by utilizing a deep learning semantic segmentation network, and then eliminating dynamic characteristics in the scene by combining semantic information; 2, introducing a feature descriptor enhancement algorithm, and enhancing descriptors of front-end extraction feature points; 3, constructing key frame point clouds by using the key frames of the SLAM system, and then splicing the point clouds of the key frames to construct a global dense point cloud map; and 4, connecting semantic information obtained by semantic segmentation with the global point cloud to construct a semantic map. According to the method, through elimination of scene dynamic features, enhancement of descriptors and semantic map construction, stable tracking of an SLAM system in a dynamic scene is effectively improved, and description of environmental semantics is increased.

Description

Technical Field

[0001] This invention relates to the field of visual simultaneous localization and mapping (SLAM) technology, and to a visual SLAM method for enhancing descriptor semantics in dynamic environments. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is one of the core technologies for robot perception. However, traditional visual SLAM algorithms are often based on strong assumptions about a static environment. When moving objects are present in the scene, their localization accuracy deteriorates significantly. For example, the classic ORB-SLAM series of algorithms are mainly designed for static or rigid, immovable objects, which makes robust pose estimation in dynamic environments a serious challenge. For dynamic scenes, many algorithms in recent years have incorporated deep learning into the ORB-SLAM series to attempt to add dynamic processing, performing dynamic feature processing and then using the preserved features for matching and tracking. However, the traditional ORB descriptors used for feature preservation lack geometric information and have limitations in global perception, exhibiting insufficient robustness in complex environments such as changes in lighting and scale. In the simultaneous localization and mapping process, map semantics can be added to achieve an understanding of the semantic information of the environment.

[0003] Therefore, there is a need for a SLAM system that can perform dynamic feature processing and enhance descriptors, while also constructing semantic low-level graphs to ensure stable tracking in dynamic environments and to achieve semantic cognition of the scene. Summary of the Invention

[0004] The purpose of this method is to provide a visual SLAM method that enhances descriptor semantics in dynamic environments, aiming to solve the adaptability problem of SLAM systems in dynamic environments and have the ability to construct semantic maps.

[0005] To achieve the above objectives, this invention provides a method for enhancing descriptor semantic visual SLAM in dynamic environments, comprising:

[0006] Step 1: Use a deep learning semantic segmentation network to perform semantic segmentation on all kinds of objects in the scene. Then, based on the results of semantic segmentation, combine semantic information to perform feature elimination on the features of dynamic objects in the scene.

[0007] Step 2: Introduce a feature descriptor enhancement algorithm to enhance the traditional description and improve the quality of feature points;

[0008] Step 3: Construct a single keyframe point using the keyframes of the SLAM system and the corresponding depth images, and then construct a global dense point cloud map by stitching together the point clouds of multiple keyframes.

[0009] Step 4: Associate the semantic labels of semantic segmentation with the global point cloud to construct a global dense point cloud map with semantic information;

[0010] A further technical solution of the present invention: The semantic segmentation network used in step 1 is DeepLabV3+. DeepLabV3+ achieves pixel-level classification by extracting deep features of the image in the encoder and combining the shallow features obtained from the image with the deep features of the encoder in the decoder. Based on the semantic results obtained from the semantic segmentation network, the semantic information is matched to identify moving objects in the scene, generating a mask containing only moving objects, and using the mask for dynamic feature elimination.

[0011] A further technical solution of the present invention: In step 2, the FeatureBooster descriptor enhancement algorithm is used to enhance the traditional ORB descriptor of ORB-SLAM3. This algorithm uses two stages, self-enhancement and cross-enhancement, to enhance the descriptor.

[0012] A further technical solution of the present invention: The point cloud construction of a single key frame in step 3 is to convert the pixel coordinates of the image and the depth value of the depth map from 2D coordinates to 3D coordinates to determine the position of the point cloud in three-dimensional space. Based on the mutual stitching of the three-dimensional spatial positions of multiple frames of point clouds, the actual scene can be reconstructed in three dimensions, that is, a global dense point cloud.

[0013] A further technical solution of the present invention: In step 5, each pixel of the keyframe is combined with a semantic tag to generate the probability distribution of each pixel in each category. At the same time, the point cloud mapped by the pixel has the same probability distribution of each category. The probability of the point cloud mapped by the pixel is combined with the probability of the original point cloud and the maximum a posteriori probability is used to solve the category of the point cloud mapped by the current pixel.

[0014] In summary, this invention discloses an enhanced descriptor semantic visual SLAM method for dynamic environments. It includes using the ORB-SLAM3 algorithm to address situations where moving objects exist in indoor scenes. However, this algorithm suffers from significant trajectory drift during localization and mapping, leading to a marked decrease in localization accuracy. To solve this problem, the proposed SLAM system uses a DeepLabV3+ semantic segmentation network to identify moving objects in the scene. For the identified region, i.e., the dynamic feature region, dynamic feature removal is performed, allowing the SLAM system to utilize the remaining features for matching and tracking, achieving more accurate localization.

[0015] Traditional ORB descriptors in ORB-SLAM3 systems lack robustness to changes in lighting, blur, and repetitive textures, and their performance is limited when the scene scale changes drastically. To address these limitations, the FeatureBooster algorithm is introduced, which optimizes the ORB descriptor through a two-stage enhancement mechanism. SLAM systems using the enhanced descriptors can achieve more efficient matching and tracking.

[0016] To construct a map incorporating environmental semantic information, this SLAM system introduces a semantic mapping module. This module first uses keyframes generated by the tracking thread to build a dense point cloud map. Then, it applies the DeepLabV3+ semantic segmentation network to obtain pixel-level semantic labels and associates these labels with the point cloud to ultimately form a global semantic point cloud map. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the present invention.

[0018] Figure 2 Flowchart of semantic segmentation network mask generation.

[0019] Figure 3 Feature removal flowchart.

[0020] Figure 4 Descriptor-enhanced network graph.

[0021] Figure 5 ORB descriptor enhancement flowchart.

[0022] Figure 6 Semantic map construction flowchart Detailed Implementation

[0023] Embodiments of the method are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the method, and should not be construed as limiting the method.

[0024] Please see Figure 1 This invention provides a method for enhancing descriptor semantic visual SLAM in dynamic environments, comprising the following steps:

[0025] Step 1: Use a deep learning semantic segmentation network to perform semantic segmentation on all kinds of objects in the scene. Then, based on the results of semantic segmentation, combine semantic information to perform feature elimination on the features of dynamic objects in the scene.

[0026] Specific methods:

[0027] like Figure 2As shown, the DeepLabV3+ semantic segmentation network is an encoder-decoder structure. The core of the encoder lies in expanding the receptive field while maintaining the feature map resolution through dilated convolution, and fusing multi-scale and global features through an adjustable output stride combined with the dilated spatial pyramid pooling (ASPP) module, so as to output a high-resolution feature map rich in semantic information for the encoder, achieving accurate pixel-level classification.

[0028] In the decoder, the encoder features are first bilinearly upsampled by a factor of 4, and then concatenated and fused with low-level features of the same resolution as the backbone network. To prevent the low-level features from having too many channels and suppressing the semantic information of the encoder, they are first reduced in dimensionality using 1x1 convolutions. The concatenated features are then refined and optimized through several 3x3 convolutions, and finally upsampled by a factor of 4 to output the prediction. This method can recover the details of object boundaries.

[0029] This article focuses on indoor scenarios. Figure 2 In this process, the original image is processed by a semantic segmentation network to obtain a labeled mask image. Then, based on the semantic information of the moving objects (people), a mask is generated for potential moving objects. The mask is used to remove dynamic features of the scene. The process is as follows: Figure 3 As shown.

[0030] Step 2: Introduce a feature descriptor enhancement algorithm to enhance the traditional description and improve the quality of feature points;

[0031] Specific methods:

[0032] like Figure 4 As shown, the FeatureBooster algorithm is implemented in two steps: self-boosting and cross-boosting. The self-boosting module uses a lightweight MLP to map the original features to a new space and fuses the geometric attributes of key points (position, score, orientation / scale) to optimize the descriptor.

[0033] The cross-enhancement module uses a lightweight Transformer to enhance individual descriptors by leveraging descriptors from other features and the spatial layout of all features. The entire network is trained end-to-end, and the loss function combines retrieval loss and enhancement loss.

[0034] Changing the similarity metric is equivalent to mapping (projecting) the original descriptor to a new space, which improves matching performance. The self-enhancement stage uses a multilayer perceptron (MLP) to learn an approximate optimal projection function, as shown below:

[0035]

[0036] in Indicates the first Descriptors for key points, This represents the descriptor after the projection transformation.

[0037] To enhance the utilization of keypoint geometric information, the descriptor algorithm encodes 2D position, scale, orientation, and detection score. Similarly, the MLP embeds this geometric information into a high-dimensional vector, further improving the descriptor. Geometric information is used... This indicates that, among them, the position , test score ,scale ,direction The mapping function is as follows:

[0038]

[0039] To address the issue that self-enhancement only focuses on local upper and lower constraints, the enhanced descriptor algorithm, during the cross-enhancement stage, utilizes a lightweight Transformer attention mechanism to aggregate the spatial context of sparse local features in the same image, forming a global context to enhance the descriptor. This expands the receptive field and enables it to perceive the context of neighboring features, significantly improving discriminability, especially for features in repetitive structures. The projection description is as follows:

[0040]

[0041] Where, in the formula In the image Local features.

[0042] In the feature extraction process of the ORB-SLAM3 system, the ORB descriptor is represented as 256-bit binary data. For example... Figure 5 As shown, in the implementation of the enhanced descriptor algorithm, the 256-bit binary data is first mapped to floating-point values ​​to meet the input requirements of the neural network. , Floating-point mapping The formula is as follows:

[0043]

[0044] in .

[0045] After floating-point mapping, the descriptors are enhanced by using self-enhancement and cross-enhancement algorithms to improve the performance of the given descriptors and increase their correlation with global descriptors, thereby effectively improving the performance of subsequent feature matching.

[0046] Step 3: Construct a single keyframe point using the keyframes of the SLAM system and the corresponding depth images, and then construct a global dense point cloud map by stitching together the point clouds of multiple keyframes.

[0047] Specifically, such as Figure 6 As shown, when a keyframe is input into the semantic map construction task, the first step is to construct its own 3D point cloud. The point cloud construction of a single frame is achieved by transforming the pixel coordinates of the image with the depth values ​​of the depth map, converting 2D coordinates (pixel coordinates) to 3D coordinates (camera coordinates). Then, based on the pose of the keyframe, the camera coordinates are transformed to world coordinates, thus achieving the construction of a 3D point cloud in the world coordinate system. Each generated keyframe point cloud is then stitched together with the already generated point clouds to achieve the goal of constructing a global point cloud.

[0048] Step 4: Associate the semantic labels of semantic segmentation with the global point cloud to construct a global dense point cloud map with semantic information;

[0049] Specifically, a semantic segmentation network is used to obtain semantic labels from keyframes and associate them with the global point cloud to create a 3D point cloud with semantic information. Each pixel in the keyframe is combined with a semantic label, generating the probability distribution of each pixel in each category. The global point cloud is then projected onto the current keyframe. The probability distribution is represented by a vector, where each element represents the probability of each class. The semantic label for each pixel is determined using Bayes' maximum a posteriori probability formula, as follows:

[0050]

[0051]

[0052] Where Pm represents the prior probability of the original point cloud in each class, Pc represents the probability of the pixel projected onto the current frame in each class, c represents the number of classes trained by the semantic segmentation network, Pmax represents the posterior probability, and S lable This represents the vector position corresponding to the maximum a posteriori probability. According to S... lable By mapping semantic labels, and then projecting the global point cloud from the previous point cloud, we can obtain a point cloud associated with the corresponding semantic labels.

Claims

1. A method for enhancing descriptor semantics in a dynamic environment using visual SLAM, characterized in that, Includes the following steps: Step 1: Use a deep learning semantic segmentation network to perform semantic segmentation on all kinds of objects in the scene. Then, based on the results of semantic segmentation, combine semantic information to perform feature elimination on the features of dynamic objects in the scene. Step 2: Introduce a feature descriptor enhancement algorithm to enhance the traditional description and improve the quality of feature points; Step 3: Construct a single keyframe point using the keyframes of the SLAM system and the corresponding depth images, and then construct a global dense point cloud map by stitching together the point clouds of multiple keyframes. Step 4: Associate the semantic labels of semantic segmentation with the global point cloud to construct a global dense point cloud map with semantic information.

2. The method for enhancing descriptor semantic visual SLAM in a dynamic environment as described in claim 1, characterized in that... The semantic segmentation network used in step 1 is DeepLabV3+. DeepLabV3+ achieves pixel-level classification by extracting deep features from the image in the encoder and combining the shallow features obtained from the image with the deep features from the encoder in the decoder. Based on the semantic results obtained from the semantic segmentation network, the semantic information is matched to identify moving objects in the scene, generating a mask containing only moving objects, and using the mask for dynamic feature removal.

3. The method for enhancing descriptor semantic visual SLAM in a dynamic environment as described in claim 1, characterized in that... In step 2, the FeatureBooster descriptor enhancement algorithm is used to enhance the traditional ORB descriptor of ORB-SLAM3. This algorithm uses two stages, self-enhancement and cross-enhancement, to enhance the descriptor.

4. The method for enhancing descriptor semantic visual SLAM in a dynamic environment as described in claim 1, characterized in that... In step 3, the point cloud construction of a single keyframe is achieved by converting the pixel coordinates of the image to 3D coordinates using 2D coordinates and the depth values ​​of the depth map, thereby determining the position of the point cloud in 3D space. By stitching together the 3D spatial positions of multiple point clouds, a 3D reconstruction of the actual scene can be realized, i.e., a globally dense point cloud.

5. The method for enhancing descriptor semantic visual SLAM in a dynamic environment as described in claim 1, characterized in that... In step 4, each pixel of the keyframe is combined with a semantic label to generate the probability distribution of each pixel in each category. At the same time, the point cloud mapped by the pixel has the same probability distribution of each category. The probability of the point cloud mapped by the pixel is combined with the probability of the original point cloud and the maximum a posteriori probability is used to solve the category of the point cloud mapped by the current pixel.

Images