Dynamic scene visual SLAM positioning method based on geometric constraint
By combining feature extraction, YOLOv8 object detection, sparse optical flow graph clustering, and multi-view geometry algorithms, dynamic object feature points are eliminated, solving the problem of inaccurate localization in dynamic scenes by visual SLAM and achieving higher accuracy and robustness.
Patent Information
- Application Number
- CN202510643978.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-19
- Publication Date
- 2025-11-14
AI Technical Summary
Existing visual SLAM technology is easily affected by moving objects in dynamic scenes, resulting in inaccurate localization and map updates, and poor robustness.
A geometric constraint-based approach is adopted, combining a feature extraction module, a YOLOv8 object detection module, a sparse optical flow graph clustering algorithm, a multi-view geometric algorithm, and a depth-based mask dilation algorithm to remove feature points of dynamic objects and perform inter-frame tracking and mapping.
It improves the localization accuracy and robustness of visual SLAM in dynamic scenes, reduces localization error, and significantly improves the recognition rate and initialization time of traditional methods in dynamic environments.
Smart Images
Figure CN120947622A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of dynamic visual positioning technology, and in particular to an indoor visual SLAM positioning method based on geometric constraints. Background Technology
[0002] Visual Simultaneous Localization and Mapping (SLAM) is a key technology for achieving environmental mapping and self-localization using visual sensors. As a core research direction in robotics and computer vision, it has broad application value in fields such as autonomous driving, augmented reality (AR), virtual reality (VR), drone navigation, service robots, and intelligent manufacturing. Visual SLAM technology acquires high-precision spatial information and pose estimation through visual data captured by cameras, without relying on expensive LiDAR equipment, offering advantages such as low cost and high flexibility. With the rapid development of deep learning and hardware computing power, visual SLAM is continuously overcoming robustness and real-time limitations in complex scenes, demonstrating broad application prospects and providing important technical support for achieving intelligent and autonomous systems.
[0003] Visual SLAM assumes a static scene, but moving objects in dynamic scenes (such as pedestrians, vehicles, and animals) significantly interfere with the localization and mapping process. These dynamic objects can lead to mismatched feature points, inaccurate map updates, and trajectory drift, significantly reducing the robustness and accuracy of the SLAM system. To mitigate the interference of dynamic scenes, researchers have proposed various solutions, including applying deep learning techniques such as semantic segmentation and object detection, combined with semantic information for scene understanding. The combination of these techniques has played a crucial role in improving the stability of visual SLAM in dynamic environments. However, methods based on deep learning to acquire semantic information about dynamic objects are only applicable to specific prior objects in the environment, and deep learning methods may suffer from missed recognition issues. Therefore, it is necessary to design geometric constraint algorithms based on the geometric constraints of feature points in images, and combine them with deep learning methods to achieve a visual SLAM system with stronger resistance to dynamic interference. Summary of the Invention
[0004] The problem this invention aims to solve is how to provide a visual SLAM method that can better handle complex dynamic scenes.
[0005] To address the aforementioned technical problems, the present invention provides a dynamic scene visual SLAM localization method based on geometric constraints, characterized by the following steps:
[0006] A dynamic scene visual SLAM localization method based on geometric constraints is characterized by comprising a feature extraction module, a YOLOv8 object detection module, a sparse optical flow graph clustering algorithm, a multi-view geometry algorithm, a depth-based mask dilation, tracking, and mapping. The feature extraction module extracts feature points from the image; the YOLOv8 object detection module identifies prior dynamic objects in the image; the sparse optical flow graph clustering algorithm identifies feature points on the dynamic objects based on the preliminary detection results of the YOLOv8 object detection module; the multi-view geometry algorithm, as a supplement to the sparse optical flow graph clustering algorithm, improves the overall recognition rate; the depth-based mask dilation algorithm generates a mask covering the dynamic objects based on the recognition results of the sparse optical flow graph clustering algorithm and the multi-view geometry algorithm, which is used to remove all feature points on the dynamic objects; the tracking and mapping module uses static feature points for inter-frame tracking, local map maintenance, and loop closure detection to ultimately achieve localization and mapping in the dynamic scene.
[0007] Furthermore, the feature extraction module is used to extract ORB feature points in the image. First, it uses the FAST keypoint detector to extract FAST keypoints in the image, then calculates the BRIEF descriptor for each keypoint, and finally completes the extraction of ORB feature points.
[0008] Furthermore, the YOLOv8 object detection module uses the YOLOv8n model with the smallest parameter size to enable deployment and application on computationally limited platforms such as robots. The model was pre-trained on the COCO dataset and can recognize 80 common objects in daily life. The YOLOv8 object detection module processes the input images in real time, using anchor boxes to label the positions of prior dynamic objects in the images.
[0009] Furthermore, the sparse optical flow graph clustering algorithm includes steps such as static optical flow information calculation, KNN-based graph clustering, and subgraph motion determination. The static optical flow information calculation step is used to obtain the mean and covariance of the static optical flow; the KNN-based graph clustering step is used to find point clusters (subgraphs) with similar optical flow information in the image; the subgraph motion determination step determines the motion of each point cluster to obtain reliable dynamic feature points.
[0010] Furthermore, the implementation process of calculating the static optical flow information includes: firstly, performing LK sparse optical flow on the feature points in the image; then, calculating the average value and covariance matrix of the optical flow vectors for all feature points whose optical flow results fall outside the target detection anchor box, thereby obtaining the static optical flow information.
[0011]
[0012] Furthermore, the KNN-based graph clustering process includes: first, applying the KNN clustering algorithm to all feature points in the image according to the distance between pixels to obtain an undirected graph where feature points that are close to each other are connected by edges; then, clipping the edges based on the difference in optical flow and depth between the two vertices of the edge; and finally, clipping the edges when the Euclidean distance between the optical flow vectors of the two vertices is greater than a threshold τ. v Or the depth difference is greater than τ d Cut this edge:
[0013]
[0014] |d1-d2|>τ d (4)
[0015] After cropping, several sub-images can be obtained. The feature points contained in these sub-images can be considered to have similar optical flow and depth properties.
[0016] Furthermore, the subgraph motion determination method combines the methods based on optical flow anomaly analysis and epipolar constraint. If both methods consistently determine that the subgraph is dynamic, then the subgraph is considered to be in a dynamic state.
[0017] First, an optical flow anomaly analysis method is used to calculate the average optical flow value of each vertex of each subgraph. And calculate the chi-square test value between it and the static optical flow:
[0018]
[0019] Set threshold When the chi-square value exceeds the threshold, the epipolar constraint technique is further used to determine the motion of the subgraph; otherwise, the subgraph is considered static, and the next subgraph is examined. The process based on the epipolar constraint method includes: for a pair of correctly matched feature points P1 and P2 in adjacent images, on the normalized plane, P1 = [u1, v1, 1] T P2 = [u2, v2, 1] T Let F represent the coordinates of two points, and let F represent the fundamental matrix calculated from the feature points outside the anchor frame; then we have the epipolar line l2: τ F
[0020]
[0021] The distance D from feature point P2 to epipolar line l2 is:
[0022]
[0023] For static feature points, D approaches 0, therefore an epipolar distance threshold τ is set. F When D > τ FAt that time, it was finally determined that the feature points were in a dynamic state.
[0024] Furthermore, the multi-view geometry algorithm first performs inter-frame tracking on the current frame and extracts keyframes with similar positions; then, for each keyframe KF, it projects the feature points on it onto the current frame CF and records them as , and calculates the projection depth; calculates the angle between and to the common observation point in 3D space; obtains the measured depth value at based on the depth map of the RGBD camera, and calculates the depth difference; if and , the point is considered static, otherwise it is considered dynamic.
[0025] Furthermore, the depth-based mask expansion algorithm utilizes the depth map of the RGBD camera to expand the extracted dynamic points pixel by pixel. When the depth value of a dynamic point is similar to that of its neighboring pixel, the neighboring pixel is considered to be dynamic as well, thereby gradually forming a mask that covers the entire dynamic object.
[0026] Furthermore, in the tracking and mapping process, all feature points outside the mask of dynamic objects are identified as static feature points, and these are sent back to the tracking thread for subsequent local mapping, loop closure detection, map construction, and other functions.
[0027] The beneficial effects of this invention are as follows:
[0028] (1): The sparse optical flow graph clustering algorithm of the present invention has the effect of improving the recognition rate of dynamic objects by geometric constraint algorithms, and solves the problems of long initialization time and low recognition rate of slow and rotating objects in traditional multi-view geometric algorithms.
[0029] (2): The dynamic scene visual SLAM localization method of the present invention has the effect of reducing localization error in dynamic environment and achieves better localization effect compared with traditional dynamic scene SLAM algorithm. Attached Figure Description
[0030] To more clearly illustrate the implementation of the present invention or the existing technical solutions, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.
[0031] Figure 1 A flowchart illustrating the steps of a dynamic scene visual SLAM localization method based on geometric constraints provided for the invention;
[0032] Figure 2 A flowchart illustrating the steps of the sparse optical flow graph clustering algorithm in a dynamic scene visual SLAM localization method based on geometric constraints provided for the invention. Detailed Implementation
[0033] The present invention will be described below with reference to specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.
[0034] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.
[0035] Reference Figure 1 This invention discloses a visual SLAM localization method in dynamic scenes, comprising the following steps:
[0036] S101: Extract ORB feature points from the input RGB image;
[0037] S102: Detect potential dynamic objects in an image using the YOLOv8 object detection algorithm.
[0038] S103: Extract dynamic points in an image using a sparse optical flow graph clustering algorithm.
[0039] S104: Further extract dynamic points from the image using a multi-view geometry algorithm.
[0040] S105: Using the dynamic points extracted in S103 and S104 as "seed points", a mask covering the entire dynamic object is obtained by using a mask expansion algorithm based on depth information.
[0041] S106: Feature points not covered by the dynamic object mask are considered static feature points. Static feature points are used for subsequent tracking, local mapping, loop closure detection, and other threads.
[0042] As a technical optimization scheme of the present invention, the sparse optical flow graph clustering algorithm is used to improve the shortcomings of the traditional algorithm in terms of insufficient dynamic point recognition rate, and includes the following specific steps, such as... Figure 2 As shown:
[0043] First, LK sparse optical flow is applied to the feature points in the image.
[0044] Then, for all feature points whose optical flow results fall outside the target detection anchor box, the mean and covariance matrix of their optical flow vectors are calculated to obtain static optical flow information:
[0045]
[0046] Then, the KNN clustering algorithm is applied to all feature points in the image according to the distance between pixels to obtain an undirected graph in which feature points that are close to each other are connected by edges.
[0047] Edges are clipped based on the difference in optical flow and depth between their two vertices. Clipping occurs when the Euclidean distance between the optical flow vectors of the two vertices is greater than a threshold τ. v Or the depth difference is greater than τ d Cut this edge:
[0048]
[0049] |d1-d2|>τ d (4)
[0050] After cropping, several sub-images can be obtained. The feature points contained in these sub-images can be considered to have similar optical flow and depth properties.
[0051] Finally, for each subgraph, the average optical flow value at its vertices is calculated, and the chi-square test value between it and the static optical flow is calculated:
[0052]
[0053] Set threshold When the chi-square value exceeds the threshold, the epipolar constraint technique is further used to determine the motion of the subgraph, and an epipolar distance threshold τ is set. F When D > τ F At that time, the feature points are identified as being in a dynamic state.
[0054] To evaluate the accuracy of this method in dynamic indoor scenes, it was tested using the TUM RGB-D dataset. This dataset contains synchronized RGB and depth images and provides real-world data on camera positions, facilitating the evaluation of algorithm performance. The walking sequences in the TUM dataset are commonly used test sequences for dynamic scene SLAM. In this set of sequences, two pedestrians move in front of the camera, and the scene also includes common indoor objects such as tables, chairs, and computers.
[0055] The performance of visual SLAM systems is typically evaluated using absolute trajectory error (ATE) and relative pose error (RPE). Absolute trajectory error is calculated by aligning the estimated trajectory with the ground truth trajectory using a rigid body transformation, and then calculating the Euclidean distance between the estimated and ground truth poses at each time point. It is generally used to measure the global consistency between the estimated and ground truth trajectories. Relative pose error, on the other hand, involves selecting multiple time points within a fixed time interval Δt, calculating the pose change within each time interval, and then determining the translation and rotation errors between each pair of incremental poses. Relative pose error is primarily used to evaluate the local consistency between the estimated and ground truth trajectories.
[0056] The proposed method was compared with the ORBSLAM3 algorithm. The experimental results were the average of five experiments. The performance improvement of the proposed method in terms of absolute trajectory error compared with the ORBSLAM3 algorithm was calculated using the following formula.
[0057]
[0058] Experimental results demonstrate that this method effectively ensures the accuracy and robustness of visual SLAM localization in dynamic scenes. Compared with the ORBSLAM3 algorithm, the localization accuracy of this method can be improved by up to 97.3%, significantly enhancing the ability of the visual SLAM system to cope with complex dynamic scenes. In terms of dynamic point recognition, the sparse optical flow graph clustering algorithm effectively improves the shortcomings of traditional multi-view geometric algorithms, such as long initialization time and low recognition rate, ensuring the stability and accuracy of SLAM localization.
[0059] It should be understood that the application of the invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A dynamic scene visual SLAM localization method based on geometric constraints, characterized in that, The system includes a feature extraction module, a YOLOv8 object detection module, a sparse optical flow graph clustering algorithm, a multi-view geometry algorithm, a depth-based mask dilation algorithm, tracking, and mapping. The feature extraction module extracts feature points from the image; the YOLOv8 object detection module identifies prior dynamic objects in the image; the sparse optical flow graph clustering algorithm identifies feature points on the dynamic objects based on the preliminary detection results of the YOLOv8 object detection module; the multi-view geometry algorithm, as a supplement to the sparse optical flow graph clustering algorithm, improves the overall recognition rate; and the depth-based mask dilation algorithm generates a mask covering the dynamic objects based on the recognition results of the sparse optical flow graph clustering algorithm and the multi-view geometry algorithm, thereby removing all feature points on the dynamic objects. The tracking and mapping module uses static feature points for inter-frame tracking, local map maintenance, and loop closure detection to ultimately achieve localization and mapping in dynamic scenes.
2. The dynamic scene visual SLAM localization method based on geometric constraints as described in claim 1, characterized in that, The feature extraction module is used to extract ORB feature points in the image. First, it uses the FAST keypoint detector to extract FAST keypoints in the image, then calculates the BRIEF descriptor for each keypoint, and finally completes the extraction of ORB feature points.
3. The dynamic scene visual SLAM localization method based on geometric constraints as described in claim 1, characterized in that, The YOLOv8 object detection module uses the YOLOv8n model with the smallest parameter size to enable deployment and application on computationally limited platforms such as robots. The model was pre-trained on the COCO dataset and can recognize 80 common objects in daily life. The YOLOv8 object detection module processes the input images in real time and uses anchor boxes to label the positions of prior dynamic objects in the images.
4. The dynamic scene visual SLAM localization method based on geometric constraints as described in claim 1, characterized in that, The sparse optical flow graph clustering algorithm includes steps such as static optical flow information calculation, KNN-based graph clustering, and subgraph motion judgment. The static optical flow information calculation step is used to obtain the mean and covariance of the static optical flow; the KNN-based graph clustering step is used to find point clusters (subgraphs) with similar optical flow information in the image; the subgraph motion judgment step judges the motion of each point cluster to obtain reliable dynamic feature points.
5. The dynamic scene visual SLAM localization method based on geometric constraints as described in claim 4, characterized in that, The implementation process of calculating the static optical flow information includes: First, LK sparse optical flow is performed on the feature points in the image. Then, for all feature points whose optical flow results fall outside the target detection anchor box, the average value and covariance matrix of their optical flow vectors are calculated to obtain static optical flow information.
6. The dynamic scene visual SLAM localization method based on geometric constraints as described in claim 4, characterized in that, The KNN-based graph clustering process includes: First, KNN clustering is applied to all feature points in the image based on pixel distances to obtain an undirected graph. In this graph, feature points with similar distances are connected by edges. Then, edges are clipped based on the difference in optical flow and depth between the two vertices. Clipping occurs when the Euclidean distance between the optical flow vectors of two vertices exceeds a threshold τ. v Or the depth difference is greater than τ d Cut this edge: |d1-d2|>τ d (4) After cropping, several sub-images can be obtained. The feature points contained in these sub-images can be considered to have similar optical flow and depth properties.
7. The dynamic scene visual SLAM localization method based on geometric constraints as described in claim 4, characterized in that, The process of determining the motion of a subgraph includes methods based on optical flow anomaly analysis and methods based on epipolar constraints. If both methods consistently determine that the subgraph is dynamic, then the subgraph is considered to be in a dynamic state. First, an outlier-based method based on optical flow analysis is used for judgment: for each subgraph, the average optical flow value of its vertices is calculated. And calculate the chi-square test value between it and the static optical flow: Set threshold When the chi-square value exceeds the threshold, the epipolar constraint technique is further used to determine the motion of the subgraph; otherwise, the subgraph is considered static, and the next subgraph is examined. The process based on the epipolar constraint method includes: For a correctly matched pair of feature points P1 and P2 in adjacent images, on the normalized plane, P1 = [u1, v1, 1] T P2 = [u2, v2, 1] T Let F represent the coordinates of two points, and F represent the fundamental matrix calculated from feature points outside the anchor frame; then we have the epipolar line l2: The distance D from feature point P2 to epipolar line l2 is: For static feature points, D approaches 0, therefore an epipolar distance threshold τ is set. F When D > τ F At that time, it was finally determined that the feature points were in a dynamic state.
8. The dynamic scene visual SLAM localization method based on geometric constraints as described in claim 1, characterized in that, A multi-view geometry algorithm is introduced to improve the recognition rate of dynamic points. First, inter-frame tracking is performed on the current frame, and keyframes with similar positions are extracted. Then, for each keyframe KF, the feature point x on it is projected onto the current frame CF and denoted as x′, and the projection depth z is calculated. proj ; Calculate the angle α between x and x′ to the common observation point X in 3D space; Obtain the measured depth value z′ at x′ based on the depth map from the RGBD camera, and calculate the depth difference Δz = z proj -z′; when α < 30° and Δz < 40cm, the point is considered static; otherwise, it is considered dynamic.
9. The dynamic scene visual SLAM localization method based on geometric constraints as described in claim 1, characterized in that, The depth-based mask expansion algorithm uses the depth map of an RGBD camera to expand the extracted dynamic points pixel by pixel. When the depth value of a dynamic point is similar to that of its neighboring pixel, the neighboring pixel is considered to be dynamic as well, thereby gradually forming a mask that covers the entire dynamic object.
10. The dynamic scene visual SLAM localization method based on geometric constraints as described in claim 1, characterized in that, The tracking and mapping process identifies all feature points outside the mask of dynamic objects as static feature points and sends them back to the tracking thread for subsequent local mapping, loop closure detection, map building, and other functions.