Dynamic scene VSLAM optimization method based on target detection and SuperPoint
By combining the SuperPoint and Yolo-FastestV2 algorithms, interference feature points from dynamic objects are eliminated, solving the pose estimation and positioning accuracy problems of traditional visual SLAM systems in complex dynamic environments, and achieving higher positioning accuracy and system robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-25
- Publication Date
- 2026-03-31
AI Technical Summary
Traditional visual SLAM systems are prone to feature point matching errors in complex dynamic environments, which leads to a decrease in pose estimation and localization accuracy, makes it impossible to build a globally consistent map, and affects the robustness of the system.
By combining the improved SuperPoint feature extraction algorithm with the Yolo-FastestV2 dynamic feature point removal algorithm, and through a lightweight network architecture and epipolar geometry constraint method, interfering feature points on dynamic objects are removed, thereby improving feature extraction capability and localization accuracy.
It significantly improves pose estimation and localization accuracy in complex and dynamic scenes, enhances the environmental adaptability of the visual SLAM system, and avoids the influence of dynamic objects on the system.
Smart Images
Figure CN121767441A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of visual SLAM technology, specifically relating to a dynamic scene VSLAM optimization method based on object detection and SuperPoint. Background Technology
[0002] The rapid development of artificial intelligence technology and the growing demand for intelligent products have promoted the vigorous development of robotics technology in my country. However, in some complex factory environments, the GPS function of intelligent mobile robots cannot be used. To address this issue, researchers have proposed the Simultaneous Localization and Mapping (SLAM) method.
[0003] Visual SLAM is crucial for mobile robots, combining visual perception and localization technologies to enable robots to simultaneously locate themselves and build an environmental map without prior maps. Traditional visual SLAM systems typically operate under static environmental assumptions. However, when dynamic feature points are introduced into a more complex dynamic environment, errors arise in feature point matching and initial pose estimation. These errors accumulate during the subsequent nonlinear optimization process, causing trajectory drift and preventing the construction of a globally consistent map, thus affecting the system's robustness.
[0004] Currently, mainstream visual SLAM systems mainly rely on manually selected point features for image matching and tracking. This can easily lead to tracking failures when faced with varying viewpoints, distances, and lighting conditions.
[0005] Therefore, there is an urgent need in this field for a visual SLAM method that can significantly improve pose estimation and positioning accuracy in complex dynamic scenes. Summary of the Invention
[0006] The purpose of this invention is to address the problems existing in the above-mentioned technical field by providing a dynamic scene VSLAM optimization method based on object detection and SuperPoint. By combining the improved SuperPoint feature extraction algorithm with the dynamic feature point removal algorithm based on Yolo-FastestV2, the feature extraction capability can be effectively enhanced and dynamic interference feature points in the detection box can be removed, significantly improving the pose estimation and localization accuracy of the visual SLAM system in complex dynamic scenes.
[0007] The dynamic scene VSLAM optimization method based on object detection and SuperPoint includes the following steps:
[0008] Step 1: Use the SuperPoint feature extraction algorithm as the feature extraction module, and use the lightweight network architecture MobileNetV3 with fewer parameters to lighten the encoding layer of the traditional SuperPoint;
[0009] Step 2: Introduce the YOLO-FastestV2 prior dynamic target recognition and detection module, which runs in parallel with Step 1, to extract feature points of the dynamic target region, and its backbone network adopts the lightweight ShuffleNetV2 architecture.
[0010] Step 3: Perform image detection on the real-time acquired images to generate detection boxes, and use the epipolar geometric constraint method to remove interfering feature points on dynamic objects to complete the extraction of static feature points, which are used for subsequent feature point matching and initial pose estimation.
[0011] Step 4: Use the TUM dataset for simulation verification to compare the dynamic object detection effect and the removal of dynamic interference feature points.
[0012] The SuperPoint feature extraction algorithm described in step 1 includes a shared encoder, a feature point decoder, and a descriptor decoder;
[0013] The shared coding layer consists of 8 convolutional layers. The first 4 layers consist of 64-channel 3×3 convolutions and non-linear ReLU function layers, with the 2nd and 4th layers followed by 2×2 max pooling. The last 4 layers consist of 128-channel 3×3 convolutions and non-linear ReLU function layers, with the 6th layer followed by 2×2 max pooling. The image is compressed from W×H×1 at input to W / 8×H / 8×128.
[0014] The feature point decoder decodes the initially acquired feature map to obtain an output with dimensions W / 8×H / 8×65; where the number of channels 65 corresponds to non-overlapping local 8×8 pixel block regions in the input image and one recycling channel representing no feature points; then the recycling channel is discarded through a Softmax operation, and finally the output is transformed from W / 8×H / 8×64 back to W×H×1 using a Reshape operation to obtain the feature point probability map;
[0015] The descriptor decoder decodes the preliminary feature map to obtain a descriptor feature map with dimensions W / 8×H / 8×256, and then performs bicubic linear interpolation and L2 normalization to obtain a dense descriptor vector of unit length (with dimensions H×W×256).
[0016] The training process of the SuperPoint feature extraction algorithm model described in step 1 can be divided into the following three stages:
[0017] Phase 1, Corner-aware pre-training: Using 3D objects composed of simple geometric shapes and key points as the dataset, this phase focuses on enhancing the model's sensitivity to geometric features such as object edges and corners.
[0018] Phase 2, Feature Point Self-Annotation: Introducing Homographic Adaptation (HA) technology, using the acquired original images, and performing random homography transformation on each image to obtain new images, which are then labeled and used for detection;
[0019] By fusing the detection results under multiple transformations and averaging them, the feature point positions become more stable under geometric transformations, resulting in the final feature point set expression:
[0020] ;
[0021] in, For the input image, This represents the function representing the initial detected feature points. Represents the entire image quilt Rotate, Indicates the image The number of different homography transformations performed;
[0022] Phase 3, Joint Training: The image after homography transformation in the previous phase is combined with the original image to form an image pair. These two types of images are then input into the network model of the SuperPoint feature extraction algorithm. During the feature extraction phase, the image pair is processed synchronously through a shared coding network, and feature point detection maps and descriptors are output respectively.
[0023] The specific process of using the lightweight network architecture MobileNetV3 to improve the coding layer of the traditional SuperPoint in step 1 is as follows:
[0024] The lightweight network architecture MobileNetV3 combines residual processing with a depthwise separable convolutional structure and employs an efficient h-swish activation function (HS function).
[0025] The lightweight network architecture MobileNetV3 embeds a lightweight Squeeze and Excitation attention mechanism module;
[0026] Several improved inverted residual modules (bneck modules) are adopted. The front end uses a 3×3 standard convolutional layer for initial feature extraction, HS function and downsampling operation with stride of 2; the subsequent bneck module alternately uses 3×3 depthwise separable convolution and adopts ReLU or HS function, embedding channel attention mechanism at key levels, and enhancing the expressive power of important features through adaptive feature recalibration.
[0027] The HS function is as follows:
[0028] ;
[0029] in, .
[0030] The ShuffleNetV2 architecture consists of two feature extraction modules, Unit1 and Unit2, in the basic unit. By fusing channel rearrangement and depthwise separable convolution techniques in the basic unit, the performance and efficiency of the neural network are improved.
[0031] The Unit1 module maintains the scale and number of channels of the input feature map during feature extraction and adopts a dual-channel parallel structure. The left channel group directly transmits the original features, while the right channel group performs channel compression, batch normalization and depth convolution in sequence.
[0032] The Unit2 module reduces the dimension of the input feature map to half of its original dimension using a 3×3 depthwise convolution with a stride of 2, while doubling the channel dimension.
[0033] The specific process of using epipolar geometry constraints to remove interfering feature points on dynamic objects is as follows:
[0034] A motion consistency test is performed on the boundary region features, and a dynamic threshold discrimination criterion is set to remove only feature points that meet the dynamic threshold condition; matching point pairs are determined by distance, and the fundamental matrix is solved using RANSAC. Solve the following:
[0035] ;
[0036] ;
[0037] ;
[0038] in, and It is a feature matching point pair. and These are the pixel coordinates of the matched point pair. For matching points The corresponding poles, polar lines The direction vector;
[0039] At this point, the feature matching points can be determined. to its near polar line distance for:
[0040] ;
[0041] The solution obtained With the set threshold Compare, if satisfied If so, it is considered a dynamic feature point that needs to be removed;
[0042] The simulation verification process using the TUM dataset described in step 4 is as follows: The TUM dataset is used and the depth information and RGB information are fused together so that the RGB information and depth information are both timestamped, and time synchronization and data alignment are performed.
[0043] This invention provides a dynamic scene VSLAM optimization method based on object detection and SuperPoint, belonging to the field of visual SLAM. This method improves the front-end visual odometry part of the traditional ORB-SLAM3 algorithm. First, the tracking thread of the front-end visual odometry part is improved by replacing the original feature extraction algorithm module with the SuperPoint feature extraction algorithm module. Furthermore, its encoding layer is lightweighted, effectively reducing the model parameter scale and accelerating the running speed while maintaining feature point detection accuracy. On this basis, a parallel object detection thread is added, introducing the Yolo-FastestV2 object detection model to detect dynamic objects in the environment. Finally, epipolar geometric constraints are used to remove interfering feature points on dynamic objects. In summary, the method of this invention can improve pose estimation and localization accuracy in complex dynamic scenes and has broad application value.
[0044] The advantages of this invention compared to the prior art are as follows:
[0045] To address the issue of decreased positioning accuracy in dynamic indoor environments, this invention, based on the existing ORB-SLAM3 system, firstly replaces the original ORB feature extraction algorithm module with the SuperPoint feature extraction algorithm model. This achieves stable feature extraction even in complex scenes. Furthermore, the encoding layer is improved to effectively reduce the model parameter magnitude and accelerate the running speed while maintaining feature point detection accuracy. Simultaneously, the YOLO-FastestV2 object detection model is introduced. Finally, epipolar geometric constraints are used to remove interfering feature points on dynamic objects. This method meets the requirements of subsequent feature point matching and initial pose estimation, while also improving the system's environmental adaptability and avoiding the impact of dynamic objects on the SLAM system. Attached Figure Description
[0046] Figure 1 This is a flowchart illustrating the overall framework of the dynamic scene VSLAM optimization method based on target detection and SuperPoint of this invention.
[0047] Figure 2 This is a simplified diagram illustrating the principle and structure of the dynamic scene VSLAM optimization method based on target detection and SuperPoint of this invention.
[0048] Figure 3 This is a schematic diagram of the MobileNetV3 network structure in the dynamic scene VSLAM optimization method based on target detection and SuperPoint of the present invention.
[0049] Figure 4 This is a schematic diagram of the network structure of two basic units in the ShuffleNetV2 architecture of the dynamic scene VSLAM optimization method based on object detection and SuperPoint of this invention; wherein Figure 4 (a) is the basic feature extraction unit Unit1 without downsampling; Figure 4 (b) is Unit2, the downsampling feature extraction module with spatial dimensionality reduction;
[0050] Figure 5 This is a schematic diagram illustrating the epipolar geometry constraint principle of the dynamic scene VSLAM optimization method based on target detection and SuperPoint of the present invention.
[0051] Figure 6 This image shows the dynamic object detection performance of the Yolo-FastestV2 object detection model in the dynamic scene VSLAM optimization method based on object detection and SuperPoint in this invention on the TUM dataset; (a frame of an image from a dynamic scene in the TUM dataset was processed).
[0052] Figure 7 This is an image showing the effect of removing dynamic interference feature points using the VSLAM optimization method for dynamic scenes based on target detection and SuperPoint according to the present invention. Detailed Implementation
[0053] Example 1:
[0054] The dynamic scene VSLAM optimization method based on object detection and SuperPoint includes the following steps:
[0055] Step 1: Replace the existing ORB feature extraction module with the SuperPoint feature extraction algorithm, and use the lightweight network architecture MobileNetV3 with fewer parameters to improve the encoding layer of the traditional SuperPoint.
[0056] Step 2: Introduce the YOLO-FastestV2 prior dynamic target recognition and detection module, which runs in parallel with Step 1, to extract feature points of the dynamic target region, and its backbone network adopts the lightweight ShuffleNetV2 architecture.
[0057] Step 3: Perform image detection on the real-time acquired image and generate detection boxes. Use epipolar geometry constraint method to remove interfering feature points on dynamic objects to complete the extraction of static feature points. The static features are used for subsequent feature point matching and initial pose estimation.
[0058] Step 4: Use the TUM dataset for simulation verification to compare the dynamic object detection effect and the removal of dynamic interference feature points.
[0059] The SuperPoint feature extraction algorithm described in step 1 includes a shared encoder, a feature point decoder, and a descriptor decoder;
[0060] See appendix Figure 2 The shared coding layer consists of 8 convolutional layers. The first 4 layers consist of 64-channel 3×3 convolutions and non-linear ReLU function layers, with the 2nd and 4th layers followed by 2×2 max pooling. The last 4 layers consist of 128-channel 3×3 convolutions and non-linear ReLU function layers, with the 6th layer followed by 2×2 max pooling. The image is compressed from W×H×1 at input to W / 8×H / 8×128, achieving 8x sampling while preserving key features.
[0061] See appendix Figure 2 The feature point decoder decodes the initially acquired feature map to obtain an output with dimensions W / 8×H / 8×65; where 65 channels correspond to non-overlapping local 8×8 pixel block regions in the input image and one recycling channel representing no feature points; then the recycling channel is discarded through a Softmax operation, and finally the output is transformed from W / 8×H / 8×64 back to W×H×1 using a Reshape operation to obtain the feature point probability map;
[0062] See appendix Figure 2 The descriptor decoder decodes the preliminary feature map to obtain a descriptor feature map with dimensions W / 8×H / 8×256, and then performs bicubic linear interpolation and L2 normalization to obtain a dense descriptor vector of unit length (with dimensions H×W×256).
[0063] The training process of the SuperPoint feature extraction algorithm model described in step 1 can be divided into the following three stages:
[0064] Phase 1, Corner-aware pre-training: Using 3D objects composed of simple geometric shapes and key points as the dataset, this phase focuses on enhancing the model's sensitivity to geometric features such as object edges and corners.
[0065] Phase 2, Feature Point Self-Annotation: Introducing Homographic Adaptation (HA) technology, using the acquired original images, and performing random homography transformation on each image to obtain new images, which are then labeled and used for detection;
[0066] By fusing the detection results under multiple transformations and averaging them, the random errors of individual transformations are reduced, making the feature point positions more stable with respect to geometric transformations, thus obtaining the final feature point set expression:
[0067] ;
[0068] in, For the input image, This represents the function representing the initial detected feature points. Represents the entire image quilt Rotate, Indicates the image The number of different homography transformations performed;
[0069] Phase 3, Joint Training: The image transformed by homography in the previous phase is paired with the original image to form an image pair. These two types of images are then input into the auxiliary image training module. Figure 2 In the network model of the SuperPoint feature extraction algorithm shown, during the feature extraction stage, image pairs are processed synchronously through a shared coding network, which outputs feature point detection maps and descriptors respectively.
[0070] The specific process of using the lightweight network architecture MobileNetV3 to improve the coding layer of the traditional SuperPoint in step 1 is as follows:
[0071] The lightweight network architecture MobileNetV3 combines residual processing with a depthwise separable convolutional structure and employs an efficient h-swish activation function; furthermore, the lightweight network architecture MobileNetV3 embeds a lightweight Squeeze and Excitation (SE) attention mechanism module.
[0072] The implementation employs multiple improved inverted residual modules (bneck modules). Initial feature extraction is performed at the front end using a 3×3 standard convolutional layer, configured with the h-swish activation function (HS function) and a downsampling operation with a stride of 2. Subsequent bneck modules alternately use 3×3 depthwise separable convolutions, controlling the feature image resolution by adjusting the stride parameter s. Furthermore, ReLU or HS functions are used, embedding a channel attention mechanism at key layers to enhance the expressive power of important features through adaptive feature recalibration.
[0073] The HS function is as follows:
[0074]
[0075] in, .
[0076] The Yolo-FastestV2 prior dynamic target recognition and detection module described in step 2 adopts the lightweight ShuffleNetV2 architecture as its backbone feature extraction network. The ShuffleNetV2 architecture consists of two feature extraction modules, Unit1 and Unit2, of the basic unit. By fusing channel rearrangement and depthwise separable convolution techniques through the basic unit, information flow between channels is ensured to enhance feature representation capabilities, thereby improving the performance and efficiency of the neural network.
[0077] See appendix Figure 4 The network structure of Unit 1 is as follows: Figure 4 As shown in (a), this module keeps the scale and number of channels of the input feature map unchanged during feature extraction. It adopts a dual-channel parallel structure, with the left channel group directly transmitting the original features and the right channel group performing channel compression, batch normalization and depth convolution in sequence.
[0078] The network structure of Unit 2 (spatial dimensionality reduction downsampling) is as follows: Figure 4 As shown in (b), this module reduces the dimension of the input feature map to half of the original by using a 3×3 depthwise convolution with a stride of 2, while doubling the channel dimension.
[0079] The specific process of using epipolar geometry constraints to remove interfering feature points on dynamic objects is as follows:
[0080] A motion consistency test is performed on the boundary region features, and a dynamic threshold discrimination criterion is set to remove only feature points that meet the dynamic threshold condition.
[0081] See appendix Figure 5 This is a schematic diagram of the principle of epipolar geometry constraint, where and For two adjacent image planes, and These are a pair of feature matching points. and It is the center of the camera. and For the corresponding pair of epipolar lines; match the quasi-static feature points of the two frames of the vector, determine the matching point pairs by distance, and solve the fundamental matrix using RANSAC. The solution process is as follows:
[0082] ;
[0083] ;
[0084] ;
[0085] in, and These are the pixel coordinates of the matched point pair. For point The corresponding poles, polar lines The direction vector;
[0086] At this point, the feature matching points can be determined. to the poles distance for:
[0087] ;
[0088] The solution obtained With the set threshold Compare, if satisfied If the value is not specified, it is considered a dynamic feature point that needs to be removed and pose estimation is no longer performed.
[0089] The simulation verification process using the TUM dataset described in step 4 is as follows: The TUM dataset is adopted and the depth information and RGB information are fused to provide a rich and diverse test environment, enabling verification and performance evaluation under various conditions. The RGB information and depth information are both timestamped to facilitate time synchronization and data alignment.
[0090] To further verify the feasibility of the method of the present invention, the dataset was fed into the model of the method of the present invention for detection, so that the images of the TUM dataset generated attached... Figure 6 The detection box shown is used to extract feature points using an improved SuperPoint feature extraction algorithm. The extracted feature points are then filtered. The epipolar geometry constraint method is used to identify and filter dynamic features within the detection box. Finally, purely static feature points are retained for pose estimation. The effect after removing dynamic feature points is shown below. Figure 7As shown, the method of the present invention has been verified to have good robustness and can be used for pose estimation and precise positioning visual recognition tasks in complex dynamic scenes.
Claims
1. A dynamic scene VSLAM optimization method based on object detection and SuperPoint, including: S1. The SuperPoint feature extraction algorithm is used as the feature extraction module, and the lightweight network architecture MobileNetV3 is used to lighten the encoding layer of the traditional SuperPoint. S2. Introduce the Yolo-FastestV2 prior dynamic target recognition and detection module that runs in parallel with S1, extract feature points of dynamic target regions, and its backbone network adopts the lightweight ShuffleNetV2 architecture. S3. Perform image detection on the real-time acquired images to generate detection boxes, and use the epipolar geometric constraint method to remove interfering feature points on dynamic objects to complete the extraction of static feature points, which are used for subsequent feature point matching and initial pose estimation. S4. Use the TUM dataset for simulation verification to compare the dynamic object detection effect and the removal of dynamic interference feature points.
2. The dynamic scene VSLAM optimization method based on target detection and SuperPoint according to claim 1, characterized in that: S1. The SuperPoint feature extraction algorithm includes a shared encoder, a feature point decoder, and a descriptor decoder; The shared encoder consists of an 8-layer convolutional structure. The first four layers consist of 64-channel 3×3 convolutions and non-linear ReLU function layers, with the second and fourth layers followed by 2×2 max pooling. The last four layers consist of 128-channel 3×3 convolutions and non-linear ReLU function layers, with the sixth layer followed by 2×2 max pooling. The image is compressed from W×H×1 at input to W / 8×H / 8×128. The feature point decoder decodes the initially acquired feature map to obtain an output with dimensions W / 8×H / 8×65. Here, the number of channels 65 corresponds to non-overlapping local 8×8 pixel block regions in the input image and one recycling channel representing no feature points. Then, the recycling channel is discarded through a Softmax operation, and finally, the output is transformed from W / 8×H / 8×64 back to W×H×1 using a Reshape operation to obtain the feature point probability map. The descriptor decoder decodes the preliminary feature map to obtain a descriptor feature map with dimensions W / 8×H / 8×256, and then performs bicubic linear interpolation and L2 normalization to obtain a dense descriptor vector of unit length.
3. The dynamic scene VSLAM optimization method based on target detection and SuperPoint according to claim 2, characterized in that: The dense descriptor vector has a dimension of H×W×256.
4. The dynamic scene VSLAM optimization method based on target detection and SuperPoint according to claim 1, 2 or 3, characterized in that: S1. The training process of the feature extraction module can be divided into the following three stages, specifically: Phase 1, Corner Awareness Pre-training: Using a 3D object composed of simple geometric shapes and key points as a dataset, train the sensitivity of geometric features; Phase 2, Feature Point Self-Annotation: Adaptive Homography Transform (HA) technology is introduced. The original images are used, and each image is randomly transformed to obtain new images. The images are then labeled and used for detection. By fusing the detection results under multiple transformations and averaging them, the feature point positions become more stable under geometric transformations, resulting in the final feature point set expression: ； in, For the input image, This represents the function representing the initial detected feature points. Represents the entire image quilt Rotate, Indicates the image The number of different homography transformations performed; Phase 3, Joint Training: The image after homography transformation in the previous phase is combined with the original image to form an image pair. These two types of images are then input into the network model of the SuperPoint feature extraction algorithm. During the feature extraction phase, the image pair is processed synchronously through a shared coding network, and feature point detection maps and descriptors are output respectively.
5. The dynamic scene VSLAM optimization method based on target detection and SuperPoint according to claim 4, characterized in that: S1. The specific process of using the lightweight network architecture MobileNetV3 to improve the coding layer of the traditional SuperPoint is as follows: The lightweight network architecture MobileNetV3 combines residual processing with a depthwise separable convolutional structure and employs the efficient h-swish activation function. The lightweight network architecture MobileNetV3 embeds a lightweight Squeeze and Excitation attention mechanism module; Several improved inverted residual modules (bneck) are adopted. The front end uses a 3×3 standard convolutional layer for initial feature extraction, HS function and downsampling operation with stride of 2; the subsequent bneck module alternately uses 3×3 depthwise separable convolution and adopts ReLU or HS function, embedding channel attention mechanism at key levels, and enhancing the expressive power of important features through adaptive feature recalibration. The HS function is: ； in, .
6. The dynamic scene VSLAM optimization method based on target detection and SuperPoint according to claim 5, characterized in that: The ShuffleNetV2 architecture consists of two feature extraction modules, Unit1 and Unit2, of the basic unit; Module Unit1 maintains the scale and number of channels of the input feature map during feature extraction. It adopts a dual-channel parallel structure, with the left channel group directly transmitting the original features and the right channel group performing channel compression, batch normalization and depthwise convolution in sequence. Module Unit2 uses a 3×3 depthwise convolution with a stride of 2 to reduce the dimension of the input feature map to half of its original dimension, while doubling the channel dimension.
7. The dynamic scene VSLAM optimization method based on target detection and SuperPoint according to claim 6, characterized in that: S3. The specific process of using the epipolar geometric constraint method to remove interfering feature points on dynamic objects is as follows: A motion consistency test is performed on the boundary region features, and a dynamic threshold discrimination criterion is set to remove only feature points that meet the dynamic threshold condition; matching point pairs are determined by distance, and the fundamental matrix is solved using RANSAC. Solve the following: ； ； ； in, and It is a feature matching point pair. and These are the pixel coordinates of the matched point pair. For matching points The corresponding poles, polar lines The direction vector; Find the feature matching points to its near polar line distance for: ； The solution obtained With the set threshold Compare, if satisfied If so, it is considered a dynamic feature point that needs to be removed.
8. The dynamic scene VSLAM optimization method based on target detection and SuperPoint according to claim 7, characterized in that: S4. The simulation verification process using the TUM dataset is as follows: the TUM dataset is used and the depth information and RGB information are fused together so that the RGB information and depth information are both timestamped, and time synchronization and data alignment are performed.