Visual SLAM method and system for dynamic environment

Abstract

The invention discloses a visual SLAM (Simultaneous Localization and Mapping) method and a visual SLAM system for a dynamic environment, which are based on an improved ORB-SLAM 3 framework, construct a five-thread parallel processing SLAM system, and respectively comprise a dynamic region segmentation thread, a tracking thread, a local mapping thread, a closed-loop detection thread and a map construction thread. A semantic segmentation branch and a pixel clustering branch which can operate simultaneously or independently are included in a dynamic region segmentation thread, and the pixel clustering branch comprises a clustering difference sub-branch and an antipolar constraint sub-branch. In the semantic segmentation branch, image frames are subjected to semantic segmentation network and optical flow processing to identify dynamic object feature points, and mismatching feature points are eliminated based on a geometric feature filtering mechanism. In the pixel clustering branch, an image frame is divided into pixel blocks, color feature changes of the pixel blocks are analyzed through K-means clustering, and a motion area is detected by using SIFT feature points and an improved RANSAC strategy in combination with epipolar geometric constraints.

Description

Technical Field

[0001] The present invention relates to a visual SLAM method and system for dynamic environments, belonging to the intersection of computer vision, deep learning and robot navigation, and is applied to scenarios such as autonomous driving, service robots and AR / VR that require high-precision dynamic scene understanding. Background Art

[0002] Visual simultaneous localization and mapping (SLAM) technology is one of the core technologies for autonomous robot navigation and environmental perception. It enables mobile robots to autonomously locate and construct environmental maps in unknown environments. Traditional visual SLAM systems, such as ORB-SLAM and LSD-SLAM, primarily rely on feature point extraction and matching to achieve camera pose estimation and environmental reconstruction. These systems perform well in static environments, achieving centimeter-level localization accuracy and high-quality mapping.

[0003] However, when these systems are applied to real-world dynamic environments, their performance degrades significantly. This is because traditional SLAM systems are designed based on a key assumption: that objects in the environment are stationary. This assumption causes the system to treat all detected feature points as part of the static environment and use them for pose calculation. However, in real-world applications, the environment often contains a large number of moving objects, such as pedestrians and vehicles. The feature points generated by these dynamic objects can interfere with the system's pose estimation, leading to deviations in trajectory calculations, ultimately affecting positioning accuracy and map quality.

[0004] Currently, improvement solutions for dynamic environments are mainly divided into two categories: geometry-based methods and semantics-based methods. Geometry-based methods, such as optical flow and scene flow analysis, detect dynamic objects by analyzing motion information between consecutive frames. These methods do not require pre-trained models and have good versatility, but are more sensitive to factors such as motion speed and lighting changes, and have high computational complexity. Semantic-based methods, on the other hand, use deep learning models (such as Mask R-CNN and YOLO) to identify dynamic objects in the scene. Although these methods have high accuracy, the model has a large amount of computational complexity, making it difficult to meet real-time requirements. They also rely on a large amount of labeled data and have limited ability to recognize unknown objects.

[0005] In addition, existing solutions have other limitations. For example, methods that rely solely on semantic segmentation cannot handle dynamic objects that have not been trained to identify, while methods that only use geometric information have difficulty distinguishing between camera motion and object motion. In terms of computational efficiency, complex semantic segmentation networks require high-performance GPU support, which limits the application of the system on resource-constrained platforms. At the same time, most existing methods use a single strategy to handle dynamic objects and lack robust solutions for multi-information fusion. The root causes of these technical limitations are that, first, the architectural design of traditional SLAM systems does not fully consider the characteristics of dynamic environments, second, there is a contradiction between the accuracy and real-time performance of dynamic object detection, and finally, the existing methods are not adaptable enough to environmental changes. These problems will limit the practicality of existing SLAM systems in complex dynamic scenes. Therefore, there is an urgent need for a new solution that can accurately identify and process various types of dynamic objects while maintaining real-time performance. Summary of the Invention

[0006] The content of this application is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this application is not intended to identify key features or essential features of the technical solution for which protection is sought, nor is it intended to limit the scope of the technical solution for which protection is sought.

[0007] In view of the problems and shortcomings in the prior art, the present invention aims to provide a visual SLAM method and system for dynamic environments. The present invention adds a dynamic segmentation thread based on the ORB-SLAM3 framework. The dynamic segmentation thread includes a semantic segmentation branch and a pixel clustering branch that can run simultaneously or independently. The semantic segmentation branch focuses on improving the filtering accuracy in a known environment, and the pixel clustering branch focuses on ensuring the filtering function in a positional environment. The interaction between the two can effectively exclude dynamic objects and improve the positioning and mapping performance in a dynamic environment. It is used to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] The present invention discloses a visual SLAM method for a dynamic environment, comprising the following steps:

[0010] Step 1: Obtain image frames and depth image frames through an RGB-D camera, and obtain ORB feature points through ORB feature extraction;

[0011] Step 2: Generate a semantic segmentation map for the image frame based on the semantic segmentation network PP-LiteSeg to identify potential moving objects;

[0012] Step 3: Use optical flow processing to calculate the target area motion distance for ORB feature points, and filter dynamic area feature points based on the semantic segmentation map;

[0013] Step 4: After filtering out the dynamic area feature points, remove abnormal feature points based on the geometric feature filtering mechanism;

[0014] Step 5: Divide the image frame into multiple pixel blocks, use K-means clustering to analyze the changes in the color features of the pixel blocks, and use SIFT feature points and RANSAC strategy with epipolar constraints to detect motion areas;

[0015] Step 6: Combine the depth mean value analysis of the depth image frame and perform regional optimization on the motion area;

[0016] Step 7: Superimpose the dynamic object feature points and dynamic areas to filter the dynamic objects, and output the remaining static objects to the subsequent thread.

[0017] Preferably, the step 3 further comprises the following steps:

[0018] Step 3.1, obtaining the ORB feature points generated by the ORB feature extractor;

[0019] Step 3.2, perform optical flow point matching on the ORB feature points between the current image frame and the adjacent image frame;

[0020] Step 3.3, use the LK optical flow method to calculate the ORB feature point displacement vector and optical flow amplitude;

[0021] In step 3.4, a threshold is set based on the statistical distribution of the optical flow amplitude to filter out dynamic area feature points.

[0022] Preferably, the step 4 further comprises the following steps:

[0023] Step 4.1, by finding the minimum Hamming distance for feature point matching, obtain abnormal feature points and matching feature points;

[0024] Step 4.2, further filtering the matching feature points based on the optimal change of the least median square LMedS estimation transformation matrix;

[0025] In step 4.3, the remaining matching feature points and their corresponding transformation matrices reflect the global motion pattern, and matching points that do not conform to the global motion pattern are eliminated.

[0026] Preferably, the step 4.1 of obtaining abnormal feature points and matching feature points by finding the minimum Hamming distance for feature point matching further includes the following steps:

[0027] Step 4.1.1, extract the adjacent image frames of the current image frame and obtain the ORB descriptor through the ORB feature extractor;

[0028] Step 4.1.2, performing preliminary matching on the ORB descriptor based on the remaining feature points after filtering out the dynamic area feature points;

[0029] Step 4.1.3, further match the feature points by finding the minimum Hamming distance;

[0030] In step 4.1.4, feature points that are not matched in the previous and next image frames are considered abnormal feature points.

[0031] Preferably, the step 4.2 of further filtering the matching feature points based on the optimal change of the least median square LMedS estimation transformation matrix further includes the following steps:

[0032] Step 4.2.1, find the transformation matrix that meets the conditions for the matched feature point pairs;

[0033] Step 4.2.2, solving the transformation matrix based on the least median square LMedS method and estimating the optimal transformation;

[0034] Step 4.2.3: further identify and filter the matched feature points based on the estimated optimal transformation.

[0035] Preferably, the step 5 of using K-means clustering to analyze the changes in the color features of the pixel blocks further includes the following steps:

[0036] Step 5.1.1, extract features from each pixel block using the K-means clustering algorithm;

[0037] Step 5.1.2, achieve effective clustering of pixel blocks by minimizing the cost function;

[0038] Step 5.1.3, for pixel blocks corresponding to adjacent continuous image frames, calculate the difference in cluster centers;

[0039] In step 5.1.4, determine whether the area is a motion area based on the difference in cluster centers.

[0040] Preferably, the step 5 of detecting the motion area using SIFT feature points and a RANSAC strategy with epipolar constraints further includes the following steps:

[0041] Step 5.2.1, randomly extract SIFT feature points from each pixel block of the current image frame;

[0042] Step 5.2.2, matching the SIFT feature points with the feature points of the corresponding image block in the previous frame;

[0043] Step 5.2.3, use epipolar constraints and fundamental matrix estimation to identify motion regions;

[0044] In step 5.2.4, all identified motion regions are integrated into the final motion region.

[0045] Preferably, the step 6 further comprises the following steps:

[0046] Step 6.1: Overlap the cluster difference sub-branch and the epipolar constraint sub-branch to obtain the black area, gray area, and white area respectively;

[0047] Step 6.2: Calculate the depth mean of the black area based on the depth map and perform depth value error matching.

[0048] Step 6.3, assimilate the white area surrounded by other areas into a gray area;

[0049] In step 6.4, change the gray area where the depth value does not change much to black, and the rest to white.

[0050] As the second aspect of the present application, the present invention also discloses a visual SLAM system for a dynamic environment, including a dynamic area segmentation thread, a tracking thread, a local mapping thread, a closed-loop detection thread and a map construction thread; the dynamic area segmentation thread includes a semantic segmentation branch and a pixel clustering branch, and the pixel clustering branch includes a clustering difference sub-branch and an epipolar constraint sub-branch; the dynamic area segmentation thread is used to perform filtering on moving objects in the environment; the tracking thread is used to locate and track posture changes and scene changes in real time to maintain map updates and camera trajectory estimation; the local mapping thread is used to realize the construction of a local point cloud map with a local key frame as the center; the closed-loop detection thread is used to detect whether a previously visited place has been visited again to realize error correction; the map construction thread is used to build and maintain the global map of the system to ensure the accuracy and real-time performance of the map.

[0051] Preferably, the semantic segmentation branch includes a semantic segmentation module, an ORB feature extraction module, an optical flow processing module and an LMEDS judgment module, and the clustering difference sub-branch includes a pixel clustering module, an epipolar constraint consistency module and a fusion optimization module; the semantic segmentation module is used to generate a semantic segmentation map based on the semantic segmentation network PP-LiteSeg; the ORB feature extraction module is used to obtain ORB feature points and ORB descriptors; the optical flow processing module is used to estimate the target area motion distance of the feature points in the calculated image frame; the LMEDS judgment module is used to further filter out abnormal feature points from the remaining feature points after filtering out the dynamic area feature points; the pixel clustering module is used to analyze the difference changes in the color features of the pixel block through K-means clustering to determine whether it is a moving area; the epipolar constraint consistency module is used to detect the moving area using SIFT feature points and the RANSAC strategy with epipolar constraints; the fusion optimization module is used to fuse and overlap the images after K-means clustering difference analysis and epipolar constraint consistency detection.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The present invention provides a visual SLAM method and system for dynamic environments. Based on the improved ORB-SLAM3 framework, the present invention constructs a SLAM system with five parallel threads, including a dynamic region segmentation thread, a tracking thread, a local mapping thread, a closed-loop detection thread, and a map building thread. The dynamic region segmentation thread includes a semantic segmentation branch and a pixel clustering branch that can run simultaneously or independently, and the pixel clustering branch includes a clustering difference sub-branch and an epipolar constraint sub-branch. The present invention can effectively filter out most of the dynamic object interference feature points by integrating the dual verification mechanism of semantic segmentation and geometric motion analysis, and significantly improve the positioning accuracy in dynamic environments. Using a lightweight PP-LiteSeg network and parallel computing architecture, the system single-frame processing time is controlled within 30ms, and the overall frame rate is stabilized at 33FPS, which can meet the needs of various real-time application scenarios and achieve efficient real-time processing performance. It also innovatively combines semantic prior knowledge and unsupervised geometric analysis, so that the system can not only accurately identify known dynamic objects (such as pedestrians), but also detect untrained moving objects, thereby enhancing the system's environmental adaptability. Effective filtering of dynamic features improves the purity of maps constructed by the system by 70%, virtually eliminating the "ghosting" phenomenon caused by dynamic objects, thereby improving map quality. Furthermore, the dual-branch redundant design ensures that even if a single module fails (such as a semantic segmentation error), the system can still maintain basic functionality through geometric analysis, enhancing system robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings constituting a part of this application are used to provide a further understanding of this application and make other features, purposes and advantages of this application more apparent. The drawings and descriptions of the exemplary embodiments of this application are used to explain this application and do not constitute an improper limitation on this application.

[0055] In the attached figure:

[0056] Figure 1 1 is a connection diagram of the main steps of the visual SLAM method in a dynamic environment according to an embodiment of the present invention;

[0057] Figure 2 1 is an overall execution diagram of the visual SLAM method in a dynamic environment according to an embodiment of the present invention;

[0058] Figure 3 This is a processing diagram for optimizing the motion area in combination with the depth image frame in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0060] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0061] The present invention discloses a visual SLAM method for dynamic environments, which will be described in detail below with reference to the accompanying drawings and in combination with embodiments. Figure 1 and Figure 2 As shown, it mainly includes the following steps:

[0062] Step 1: Obtain image frames and depth image frames through an RGB-D camera, and obtain ORB feature points through ORB feature extraction;

[0063] Step 2: Generate a semantic segmentation map for the image frame based on the semantic segmentation network PP-LiteSeg to identify potential moving objects;

[0064] Step 3: Use optical flow processing to calculate the target area motion distance for ORB feature points, and filter dynamic area feature points based on the semantic segmentation map;

[0065] Step 4: After filtering out the dynamic area feature points, remove abnormal feature points based on the geometric feature filtering mechanism;

[0066] Step 5: Divide the image frame into multiple pixel blocks, use K-means clustering to analyze the changes in the color features of the pixel blocks, and use SIFT feature points and RANSAC strategy with epipolar constraints to detect motion areas;

[0067] Step 6: Combine the depth mean value analysis of the depth image frame and perform regional optimization on the motion area;

[0068] Step 7: Superimpose the dynamic object feature points and dynamic areas to filter the dynamic objects, and output the remaining static objects to the subsequent thread.

[0069] Based on the improved ORB-SLAM3 framework, the present invention constructs a FlexiSLAM system with five parallel processing threads, including a dynamic region segmentation thread, a tracking thread, a local mapping thread, a closed-loop detection thread, and a map construction thread, forming a complete processing flow. The tracking thread, local mapping thread, closed-loop detection thread, and map construction thread are existing threads, and they interact with each other to achieve the function of synchronous positioning and mapping of the environment. The improved ORB-SLAM3 framework is parallel computing, significantly improving the real-time performance of the system. The dynamic region segmentation thread is a newly added thread in the present invention.

[0070] First, an RGB-D camera acquires image frames and depth image frames, and ORB feature points are obtained through ORB feature extraction. An RGB-D camera is a sensor that can simultaneously acquire both color RGB images of a scene and depth information. By combining a traditional color camera and depth sensor, it provides richer three-dimensional scene information than a single RGB camera. The RGB-D camera's color sensor captures visible light band information to generate regular image frames. The RGB-D camera's infrared emitter projects a random speckle pattern, and the binocular infrared camera captures the deformed pattern. The disparity is calculated through stereo matching and converted into a depth image frame. ORB feature points are generated by the ORB feature extraction module.

[0071] The system takes image and depth frames from an RGB-D camera, along with extracted ORB feature points, as input. It employs a dual-branch parallel processing mechanism, allowing the semantic segmentation branch and pixel clustering branch to run simultaneously or independently. When running simultaneously, the semantic segmentation branch focuses on improving filtering accuracy in known environments, while the pixel clustering branch focuses on ensuring filtering performance in unknown environments.

[0072] For semantic segmentation, the image frame is first fed into the lightweight semantic segmentation network PP-LiteSeg, which generates a semantic segmentation map to identify potential dynamic objects. The segmentation task is run in a separate thread in parallel with the ORB feature extractor included with the ORB-SLAM3 system to practice localization. The semantic segmentation map identifies object types at the pixel level. Potential moving objects are then identified and filtered based on the semantic information in the semantic segmentation map. Semantic information can be used to identify the pixel-level label of person or vehicle, which can be used for direct filtering, such as for people in indoor environments.

[0073] As described in step 3, the ORB feature points are processed using optical flow to calculate the target area motion distance, and the dynamic area feature points are filtered in combination with the semantic segmentation map. Specifically, the following steps are also included:

[0074] Step 3.1, obtain the ORB feature points generated by the ORB feature extractor;

[0075] Step 3.2, perform optical flow point matching on the ORB feature points between the current image frame and the adjacent image frame;

[0076] Step 3.3, use the LK optical flow method to calculate the ORB feature point displacement vector and optical flow amplitude;

[0077] In step 3.4, a threshold is set based on the statistical distribution of the optical flow amplitude to filter out feature points in dynamic areas.

[0078] Specifically, the target area motion distance is calculated by optical flow processing in combination with ORB feature points. When the motion distance exceeds the threshold, the dynamic area feature points are filtered in combination with semantic segmentation information. Specifically, the ORB feature points generated by the ORB feature extractor are obtained, and the ORB feature points between the current image frame and the adjacent image frame are matched with optical flow points. The LK (LucasKanade) optical flow method is used to calculate the feature point displacement vector (Δx, Δy) and the optical flow amplitude. The LK optical flow method is used to estimate the motion direction and speed of feature points in the image, and its optical flow amplitude is expressed as The threshold T is set based on the statistical distribution of the optical flow amplitude to filter out feature points in dynamic areas. However, the method of identifying moving objects through optical flow is simple and crude and can only be used as a supplementary method for semantic segmentation.

[0079] As described in step 4, after semantic segmentation and optical flow processing, a geometric feature-based filtering mechanism is used to remove abnormal feature points. Through feature point matching, transformation matrix solution, and global motion model consistency check, mismatched points are further eliminated to ensure pose estimation accuracy. The method of finding the minimum Hamming distance for feature point matching to obtain abnormal feature points and matching feature points specifically includes the following steps:

[0080] Step 4.1.1, extract the adjacent image frames of the current image frame and obtain the ORB descriptor through the ORB feature extractor;

[0081] Step 4.1.2, combine the remaining feature points after filtering out the dynamic area feature points to perform preliminary matching on the ORB descriptor;

[0082] Step 4.1.3, further match the feature points by finding the minimum Hamming distance;

[0083] In step 4.1.4, feature points that are not matched in the previous and next image frames are considered abnormal feature points.

[0084] Specifically, for the remaining feature points after filtering the dynamic area feature points in step 3, the ORB descriptors obtained by the ORB feature extraction module above are matched on the two frames before and after the feature points. After preliminary matching, the Hamming distance is calculated by the formula, which is expressed as:

[0085]

[0086] Among them, p′ k Represented as ORB feature points on adjacent image frames, d Hamming (d i ,d′ k ) is expressed as the Hamming distance between the ORB feature points on adjacent image frames and the remaining feature points after filtering the dynamic area, p′ j Indicates the feature point with the smallest Hamming distance. Further matching is performed on the feature points by finding the smallest Hamming distance. If the remaining feature points do not appear in the adjacent frames, they are considered abnormal feature points. The remaining feature points that appear in the adjacent frames are matched feature points.

[0087] Next, the optimal change of the transformation matrix is estimated based on the least median square LMedS to further filter the matching feature points, which specifically includes the following steps:

[0088] Step 4.2.1, find the transformation matrix that meets the conditions for the matching feature point pairs;

[0089] Step 4.2.2, solve the transformation matrix based on the least median square LMedS method to estimate the optimal transformation;

[0090] In step 4.2.3, the matching feature points are further identified and filtered based on the estimated optimal transformation.

[0091] Specifically, the remaining matched feature points are then searched for a transformation matrix that satisfies the conditions. Where M is the transformation matrix from the first frame to the second frame, ||p′ i -M·p i|| represents the Euclidean distance between the transformation point and the target point, and median(·) represents the median. Specifically, the solution of the transformation matrix M is based on the least median square (LMedS) method, which estimates the optimal transformation by minimizing the median error and can effectively resist the interference of outliers. After filtering using the least median square (LMedS) method, the retained matching points and their corresponding transformation matrix M should reflect the global motion pattern, and any matching points that do not conform to this global motion model will be eliminated. The above steps eliminate feature points that do not conform to the global motion model, which is suitable for static scenes after semantic segmentation preprocessing.

[0092] The pixel clustering branch also includes a cluster difference sub-branch and an epipolar constraint sub-branch. The cluster difference sub-branch uses K-means clustering to analyze the color feature changes of pixel blocks, and the epipolar constraint sub-branch uses SIFT feature points and an improved RANSAC strategy, combined with epipolar geometric constraints to detect motion areas. First, two consecutive frames in the image frame are divided into several 40×40 pixel blocks, that is, two adjacent consecutive frames are selected every three frames. In step 5, K-means clustering is used to analyze the changes in the color features of the pixel blocks, which specifically includes the following steps:

[0093] Step 5.1.1, extract features from each pixel block using the K-means clustering algorithm;

[0094] Step 5.1.2, achieve effective clustering of pixel blocks by minimizing the cost function;

[0095] Step 5.1.3, calculating the difference in cluster centers for pixel blocks corresponding to adjacent continuous image frames;

[0096] In step 5.1.4, determine whether the area is a motion area based on the difference in cluster centers.

[0097] For the clustering difference sub-branch, the K-means clustering algorithm is used to extract features for each pixel block. K-means clustering is a widely used unsupervised learning algorithm that can divide data into different categories based on the similarity between data points. By minimizing the formula Implement clustering, where is the i-th cluster, which contains all pixels similar to the cluster center. i is the cluster center, which represents the typical characteristics of the cluster. i || 2 is the pixel x and the cluster center μ iThe distance between them is the square of the Euclidean distance. By minimizing the cost function J, the K-means algorithm can continuously adjust the position of the cluster center so that each pixel is as close as possible to the center of the cluster to which it belongs, thereby achieving effective clustering of pixel blocks. For the corresponding pixel blocks in the two frames, calculate the difference in their cluster centers. Let μ t and μ t+1 Represent the cluster centers of the two frames respectively, through Δμ i,j =||μ t -μ t+1 The size of || determines whether the area is moving. If it is greater than a preset value (the preset value is manually set by an empirical formula), it is marked as a moving area. This method has high computational efficiency but is easily affected by light and noise.

[0098] In addition, the motion area is detected using SIFT feature points and the RANSAC strategy with epipolar constraints, which includes the following steps:

[0099] Step 5.2.1, randomly extract SIFT feature points from each pixel block of the current image frame;

[0100] Step 5.2.2, match the SIFT feature points with the feature points of the corresponding image block in the previous frame;

[0101] Step 5.2.3, use epipolar constraints and fundamental matrix estimation to identify motion regions;

[0102] In step 5.2.4, all identified motion regions are integrated into the final motion region.

[0103] For the epipolar constraint sub-branch, a SIFT feature point is randomly extracted from each pixel block of the image frame and matched with the feature point of the corresponding image block in the previous frame. The epipolar constraint and the basic matrix estimation are used to identify the motion area. The basic principle of the epipolar constraint is based on the fact that a feature point in the first frame image must be located on a specific epipolar line in the second frame image. This relationship can be expressed by the basic matrix F. The basic matrix F satisfies p′ i Fp i =0, where p′ i and p i are the homogeneous coordinates of the corresponding feature points in the two frames. This constraint indicates that the feature point p in the first frame image i Must be located at its matching point p′ i The corresponding epipolar line in the second frame is then used. Finally, all motion regions are integrated to obtain the final motion region. This step utilizes parallel processing and RANSAC filtering, resulting in high detection efficiency and robustness.

[0104] like Figure 3As shown, the image detected by K-means clustering difference analysis and RANSAC strategy with epipolar constraints is then processed based on region optimization and depth map optimization, which also includes the following steps:

[0105] Step 6.1: Overlap the images after K-means clustering difference analysis and epipolar constraint consistency detection to obtain black, gray, and white areas;

[0106] Step 6.2: Calculate the depth mean of the black area in combination with the depth image frame and perform depth value error matching;

[0107] Step 6.3, assimilate the white area surrounded by other areas into a gray area;

[0108] In step 6.4, change the gray area where the depth value does not change much to black, and the rest to white.

[0109] Specifically, the area where the cluster difference sub-branch and the epipolar constraint sub-branch overlap is identified and marked as black (i.e., the determined motion area), the area where one of the sub-branches exists is identified and marked as gray (i.e., the potential motion area), and the area where both sub-branches do not exist is identified and marked as white (i.e., the static area). The isolated areas are processed by depth-first search, and the detection accuracy is improved by combining the depth map mean analysis. The existing identified motion areas are processed, and the white areas surrounded by other areas are assimilated into gray areas. The depth mean of the black area is calculated in combination with the depth map, and the depth value error is matched. The parts of the gray area where the depth value does not change much are changed to black areas, and the others are changed to white areas. This processing improves the accuracy of motion area recognition.

[0110] Finally, the detection results (i.e., moving object masks) obtained by the semantic segmentation branch and the pixel clustering branch are superimposed to obtain the final moving object monitoring area. Targeted filtering is performed, and the remaining static objects are output to the subsequent local mapping thread to obtain a static feature point cloud. This fusion strategy retains the high precision of the semantic method while also having the generalization ability of the geometric method. In addition, the semantic segmentation branch has high detection accuracy, but has high hardware requirements and relies on training data. The pixel clustering branch does not rely on prior knowledge and is suitable for environments with limited hardware resources, but has lower accuracy. The combination of the two can complement each other's strengths. The semantic segmentation branch can quickly identify potential motion areas, and the pixel clustering branch supplements detection through frame differences, but may overfit in complex or rapidly changing environments.

[0111] The present invention also provides a visual SLAM system for a dynamic environment, including a dynamic area segmentation thread, a tracking thread, a local mapping thread, a closed-loop detection thread and a map construction thread. The dynamic area segmentation thread is a new thread added to the present invention, which is used to implement the filtering function of moving objects in the environment. The dynamic area segmentation thread is used to perform filtering on moving objects in the environment. The tracking thread is used to locate and track posture changes and scene changes in real time to maintain map updates and camera trajectory estimation. The local mapping thread is used to build a local point cloud map centered on a local key frame. The closed-loop detection thread is used to detect whether a previously visited place has been visited again to correct errors. The map construction thread is used to build and maintain the global map of the system to ensure the accuracy and real-time performance of the map.

[0112] The dynamic region segmentation thread consists of a semantic segmentation branch and a pixel clustering branch. The pixel clustering branch further includes a cluster difference sub-branch and an epipolar constraint sub-branch. The semantic segmentation branch includes a semantic segmentation module, an ORB feature extraction module, an optical flow processing module, and a LMEDS decision module. The cluster difference sub-branch includes a pixel clustering module, an epipolar constraint consistency module, and a fusion optimization module. The semantic segmentation module generates a semantic segmentation map based on the semantic segmentation network PP-LiteSeg. The ORB feature extraction module obtains ORB feature points and ORB descriptors. The optical flow processing module estimates the motion distance of the target region for feature points in the image frame. The LMEDS decision module further filters out abnormal feature points from the remaining feature points after filtering out dynamic region feature points. The pixel clustering module analyzes the differences in pixel color features using K-means clustering to determine whether a region is in motion. The epipolar constraint consistency module detects motion regions using SIFT feature points and a RANSAC strategy with epipolar constraints. The fusion optimization module fuses and optimizes the overlap of the images obtained from K-means cluster difference analysis and epipolar constraint consistency detection.

[0113] Furthermore, the improved SLAM system can be implemented on a computer equipped with an Intel i7 CPU and an NVIDIA 3060Ti GPU. The GPU accelerates semantic segmentation network computations, while the CPU handles feature extraction and geometric operations. A memory sharing mechanism enables multi-threaded data interaction, ensuring overall system efficiency. Furthermore, a frame-interval sampling strategy (processing every three frames) is employed to balance the computational load. The lightweight network and parallel architecture enable the system to achieve a processing speed of 33 FPS, meeting real-time requirements. Experimental verification demonstrates significant improvement in positioning accuracy in dynamic environments compared to ORB-SLAM3 on the TUM dataset.

[0114] In order to implement the above-mentioned embodiment, the present invention proposes an electronic device. The electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the electronic device are also stored. The processing device, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus. Typically, the following devices can be connected to the I / O interface: input devices including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices including, for example, a magnetic tape, a hard disk, etc., and communication devices. The communication device can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data.

[0115] In addition, the present invention also provides a computer storage medium. The computer storage medium can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0116] In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer storage medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof. The computer storage medium may be contained in the electronic device or may exist independently without being incorporated into the electronic device. The computer storage medium carries one or more programs that, when executed by the electronic device, enable the electronic device to implement a method and system for visual SLAM in a dynamic environment.

[0117] The above description is only an illustration of some preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A visual SLAM method for dynamic environments, characterized in that: The steps include: Step 1: Obtain image frames and depth image frames through an RGB-D camera, and obtain ORB feature points through ORB feature extraction; Step 2: Generate a semantic segmentation map for the image frame based on the semantic segmentation network PP-LiteSeg to identify potential moving objects; Step 3: Use optical flow processing to calculate the target area motion distance for ORB feature points, and filter dynamic area feature points based on the semantic segmentation map; Step 4: After filtering out the dynamic area feature points, remove abnormal feature points based on the geometric feature filtering mechanism; Step 5: Divide the image frame into multiple pixel blocks, use K-means clustering to analyze the changes in the color features of the pixel blocks, and use SIFT feature points and RANSAC strategy with epipolar constraints to detect motion areas; Step 6: Combine the depth mean value analysis of the depth image frame and perform regional optimization on the motion area; Step 7: Superimpose the dynamic object feature points and dynamic areas to filter the dynamic objects, and output the remaining static objects to the subsequent thread.

2. A visual SLAM method for a dynamic environment according to claim 1, characterized in that, The step 3 comprises the following steps: Step 3.1, obtaining the ORB feature points generated by the ORB feature extractor; Step 3.2, perform optical flow point matching on the ORB feature points between the current image frame and the adjacent image frame; Step 3.3, use the LK optical flow method to calculate the ORB feature point displacement vector and optical flow amplitude; In step 3.4, a threshold is set based on the statistical distribution of the optical flow amplitude to filter out dynamic area feature points.

3. A visual SLAM method for a dynamic environment according to claim 2, characterized in that, The step 4 comprises the following steps: Step 4.1, by finding the minimum Hamming distance for feature point matching, obtain abnormal feature points and matching feature points; Step 4.2, further filtering the matching feature points based on the optimal change of the least median square LMedS estimation transformation matrix; In step 4.3, the remaining matching feature points and their corresponding transformation matrices reflect the global motion pattern, and matching points that do not conform to the global motion pattern are eliminated.

4. A visual SLAM method for a dynamic environment according to claim 3, characterized in that, In step 4.1, abnormal feature points and matching feature points are obtained by finding the minimum Hamming distance for feature point matching, and the following steps are also included: step 4.1.1, extracting adjacent image frames of the current image frame and obtaining ORB descriptors through an ORB feature extractor; Step 4.1.2, performing preliminary matching on the ORB descriptor based on the remaining feature points after filtering out the dynamic area feature points; Step 4.1.3, further match the feature points by finding the minimum Hamming distance; In step 4.1.4, feature points that are not matched in the previous and next image frames are considered abnormal feature points.

5. A visual SLAM method for a dynamic environment according to claim 3, characterized in that, The step 4.2 further filters the matching feature points based on the optimal change of the least median square LMedS estimation transformation matrix, and further includes the following steps: step 4.2.1, searching for a transformation matrix that meets the conditions for the matched feature point pairs; Step 4.2.2, solving the transformation matrix based on the least median square LMedS method and estimating the optimal transformation; Step 4.2.3: further identify and filter the matched feature points based on the estimated optimal transformation.

6. A visual SLAM method for a dynamic environment according to claim 3, characterized in that, The step 5 adopts K-means clustering to analyze the changes of the color features of the pixel blocks, and further includes the following steps: Step 5.1.1, extract features from each pixel block using the K-means clustering algorithm; Step 5.1.2, achieve effective clustering of pixel blocks by minimizing the cost function; Step 5.1.3, for pixel blocks corresponding to adjacent continuous image frames, calculate the difference in cluster centers; In step 5.1.4, determine whether the area is a motion area based on the difference in cluster centers.

7. A visual SLAM method for a dynamic environment according to claim 6, characterized in that: In step 5, the motion region is detected using SIFT feature points and a RANSAC strategy with epipolar constraints, further comprising the following steps: Step 5.2.1, randomly extract SIFT feature points from each pixel block of the current image frame; Step 5.2.2, matching the SIFT feature points with the feature points of the corresponding image block in the previous frame; Step 5.2.3, use epipolar constraints and fundamental matrix estimation to identify motion regions; In step 5.2.4, all identified motion regions are integrated into the final motion region.

8. A visual SLAM method for a dynamic environment according to claim 7, characterized in that, The step 6 further comprises the following steps: Step 6.1: Overlap the cluster difference sub-branch and the epipolar constraint sub-branch to obtain the black area, gray area, and white area respectively; Step 6.2: Calculate the depth mean of the black area based on the depth map and perform depth value error matching. Step 6.3, assimilate the white area surrounded by other areas into a gray area; In step 6.4, change the gray area where the depth value does not change much to black, and the rest to white.

9. A visual SLAM system for dynamic environments, characterized by: The system includes a dynamic region segmentation thread, a tracking thread, a local mapping thread, a closed loop detection thread, and a map building thread that are processed in parallel; the dynamic region segmentation thread includes a semantic segmentation branch and a pixel clustering branch, and the pixel clustering branch includes a cluster difference sub-branch and an epipolar constraint sub-branch; Dynamic region segmentation thread, used to perform filtering for moving objects in the environment; The tracking thread is used to locate and track pose changes and scene changes in real time to maintain map updates and camera trajectory estimation; the local mapping thread is used to build a local point cloud map centered on a local keyframe; Closed-loop detection thread, used to detect whether a previously visited location has been revisited to implement error correction; The map building thread is used to build and maintain the global map of the system to ensure the accuracy and real-time performance of the map.

10. The visual SLAM system for dynamic environments according to claim 9, characterized in that: The semantic segmentation branch includes a semantic segmentation module, an ORB feature extraction module, an optical flow processing module and a LMEDS judgment module, and the clustering difference sub-branch includes a pixel clustering module, an epipolar constraint consistency module and a fusion optimization module; Semantic segmentation module, used to generate semantic segmentation maps based on the semantic segmentation network PP-LiteSeg; ORB feature extraction module, used to obtain ORB feature points and ORB descriptors; The optical flow processing module is used to estimate the target area motion distance of the feature points in the image frame; The LMEDS judgment module is used to further filter out abnormal feature points from the remaining feature points after filtering out the dynamic area feature points; The pixel clustering module is used to analyze the difference changes in the color features of pixel blocks through K-means clustering to determine whether they are moving areas; the epipolar constraint consistency module is used to detect moving areas using SIFT feature points and the RANSAC strategy with epipolar constraints; the fusion optimization module is used to fuse and overlap the images after K-means clustering difference analysis and epipolar constraint consistency detection.

Images