Motion segmentation based visual slam method

Abstract

The application provides a visual SLAM method based on motion segmentation, comprising the following steps: S1, for the collected RGB image, a motion segmentation method Rigidmask is used to detect potential dynamic objects and generate a dynamic object mask image, and a YOLO algorithm is used for instance segmentation to obtain an object mask image; S2, the corresponding relationship of the two mask images is matched to obtain a final mask image; S3, an ORB algorithm is used for feature extraction on the collected RGB image to obtain feature points; S4, the mask image is binarized, the feature points are matched with the obtained binarized mask image, whether the feature points are dynamic feature points is judged, if the feature points are dynamic feature points, the feature points are removed; finally, the remaining static feature points are obtained and used for subsequent pose matching and estimation. The application can remove the influence of dynamic objects on pose estimation in the visual SLAM process, reduce the interference of dynamic objects, and improve the positioning accuracy and the like.

Description

Technical Field

[0001] This invention relates to the fields of machine vision and machine learning technology, and in particular to a visual SLAM method based on motion segmentation. Background Technology

[0002] Traditional visual SLAM achieves excellent results in static scenes, where the relative positions of points of interest remain unchanged, and the only motion is caused by camera movement. However, in real-world applications, dynamic objects inevitably interfere with the scene. If only geometric constraints are used to infer scene structure, it will severely impact pose estimation accuracy. Visual SLAM algorithms for dynamic scenes need the fundamental ability to segment dynamic objects and evaluate their depth and velocity, thus removing the influence of dynamic objects on pose estimation. However, currently, no good visual SLAM algorithm can achieve this effect. Summary of the Invention

[0003] This invention aims to at least solve the technical problems existing in the prior art, and in particular, it innovatively proposes a visual SLAM method based on motion segmentation.

[0004] To achieve the above-mentioned objectives of this invention, this invention provides a motion segmentation-based visual SLAM method, comprising the following steps:

[0005] S1. For the acquired RGB image, the Rigidmask motion segmentation method is used to detect potential dynamic objects and generate dynamic object mask images. The YOLO algorithm is then used for instance segmentation to obtain the object mask images.

[0006] S2, perform correspondence matching on the two mask images to obtain the final mask image;

[0007] S3, use the ORB algorithm to extract features from the acquired RGB image to obtain feature points;

[0008] S4. Binarize the mask image, match the feature points with the obtained binarized mask image, and determine whether they are dynamic feature points. If they are dynamic feature points, remove them. Finally, obtain the remaining static feature points for pose matching and estimation.

[0009] Furthermore, the motion segmentation method Rigidmask includes:

[0010] S1-1, the initial 3D scene flow is obtained by estimating the motion of scene objects, and the camera motion is obtained by estimating the motion of the camera;

[0011] The motion estimation of the scene objects uses optical flow estimation and depth estimation methods;

[0012] The camera's motion estimation uses the NG-RANSAC algorithm, which estimates motion based on the epipolar geometry of two consecutive frames.

[0013] Due to factors such as noise and mismatch points, the obtained fundamental matrix may be unstable. Therefore, the NG-RANSAC algorithm is chosen to improve it and obtain the optimal matrix. Compared with RANSAC, NG-RANSAC requires a significantly larger number of iterations to find the minimum set without outliers. NG-RANSAC combines RANSAC with a neural network, which can predict the weight of each observation. The weights ultimately guide the sampling of the minimum set.

[0014] S1-2, Based on the initial 3D scene flow and the estimation results of camera motion, the scene flow after removing the influence of camera rotation can be obtained:

[0015]

[0016] Where Z0 and Z1 are the depths corresponding to p0 and p1, respectively;

[0017] R c This represents the camera's rotational motion, which is estimated through camera motion.

[0018] Let p0 and p1 be the homogeneous coordinates, respectively.

[0019] p0 and p1 are two corresponding points in two adjacent frames;

[0020] H R =K0R c K1 -1 Used to remove camera rotation R in 2D moving scenes c The influence of this will correct the second frame image to be in the same direction as the first frame image;

[0021] K0 and K1 represent the internal parameters of a monocular camera;

[0022] S1-3, input the scene stream into a dual-stream architecture to obtain the segmentation mask of rigid background and rigid motion instances;

[0023] In the dual-stream architecture, stream architecture 1 is used to predict whether each pixel is the background, thus obtaining a background mask; stream architecture 2 is used to predict the instance mask for each pixel, thus obtaining an instance-level image mask; then, the instance-level image mask generated by stream architecture 2 is used to refine the background mask generated by stream architecture 1, and the background mask and instance-level image mask are fitted together to filter out pixels that are inconsistent between the background mask and the instance mask.

[0024] Furthermore, the YOLO algorithm is YOLOv8, and YOLOv8 is used for instance segmentation.

[0025] Furthermore, correspondence matching is performed on the two mask images, including:

[0026] For the center point P c If the point is not within the object, use breadth-first search to find the nearest point P that has been overmasked and is closest to the center point. i As the center point; then for P i Make a judgment; if the search results are found in the Mask... yj In the middle, use Mask. yj Replace Mask ri This yields the final mask image; where Mask ri The corresponding model M r The segmentation result of the i-th image, Mask yj Model M y The segmentation result of the j-th instance.

[0027] By matching the correspondence between the mask image of the moving object obtained by Rigidmask and the mask image obtained by YOLOv8 instance segmentation, better motion segmentation results can be obtained.

[0028] Furthermore, the visual SLAM method is implemented based on ORB-SLAM.

[0029] Furthermore, the ORB-SLAM is ORB-SLAM3.

[0030] In summary, by employing the above technical solutions, this invention can remove the influence of dynamic objects on pose estimation during visual SLAM, reduce dynamic object interference, and improve positioning accuracy. Specific advantages include:

[0031] (1) A motion segmentation algorithm called Rigidmask, which combines geometric constraints, optical flow estimation and depth estimation, is introduced. The obtained dynamic object segmentation results can filter the vertices of the dynamic object prediction box and improve the accuracy of pose estimation.

[0032] (2) Based on the ORB-SLAM framework, a motion segmentation module is added, and the YOLO network is used for instance segmentation. The corresponding relationship is matched on the mask image, so that the non-rigid motion segmentation also achieves good results.

[0033] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0034] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0035] Figure 1 This is a schematic diagram of the system framework of the present invention.

[0036] Figure 2 This is a schematic diagram illustrating the segmentation error of Rigidmask for non-rigid objects. Detailed Implementation

[0037] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0038] Currently, based on different visual odometry calculation methods, computational visual SLAM is mainly divided into two categories: feature point method and direct method. The feature point method calculates camera pose and map point position based on feature point matching. PTAM (parallel tracking and mapping), proposed by Klein et al., uses FAST (features from accelerated segment test) to extract image features and was the earliest SLAM algorithm to separate tracking and mapping into two threads. ORB-SLAM2, proposed by Mur-Artal et al., uses ORB feature points for matching and is used with monocular, stereo, and RGB-D cameras, including map reuse, loop closure, and relocalization functions. ORB-SLAM3 follows the overall framework of ORB-SLAM2 and adds visual-inertial navigation, hybrid mapping, and other functions, and can run on monocular, stereo, and RGB-D cameras using pinhole or fisheye models. Direct methods do not require obtaining the correspondence between feature points; they obtain the camera pose by minimizing the photometric error. The LSD-SLAM method studied by Jakob Engel et al. is a semi-dense monocular vision SLAM, demonstrating that dense models have excellent tracking performance under rapid motion. The ORB-SLAM method used in this invention employs the feature point method.

[0039] Mainstream visual SLAM methods for dynamic scenes primarily include those based on deep learning, optical flow, and geometric constraints. Deep learning-based methods mainly utilize object detection and semantic segmentation, but learning from datasets of specific dynamic objects can lead to the exclusion of stationary, movable objects as dynamic. For example, a car parked on the roadside can still be tracked using its extracted feature points, mitigating the impact of insufficient matching feature points on pose estimation robustness. Furthermore, deep learning-based methods are overly reliant on training data; performance degrades if training data is lacking or contains homogeneous object shapes. Optical flow-based methods can detect motion information of dynamic objects without scene knowledge, but are susceptible to lighting, object occlusion, or image noise, exhibiting poor robustness. Geometric relationship-based methods are also unaffected by scene information, but due to noisy motion estimation and degenerate motion configuration, they cannot yet achieve performance comparable to other methods. This application introduces the Rigidmask method, which combines geometric constraints, optical flow estimation, and depth estimation, and employs the YOLOv8 deep learning algorithm.

[0040] To better reduce the impact of dynamic objects in the environment on pose estimation, this paper adds a motion segmentation module to ORB-SLAM3, enabling the system to more accurately segment and remove dynamic objects and adapt to localization and mapping in complex environments.

[0041] 1 System Framework

[0042] The algorithm system framework in this paper is built on the ORB-SLAM3 algorithm. As a mature SLAM system, ORB-SLAM3 has fast running speed and high stability. It is suitable for both large and small scenes, and has the functions of pure visual data processing, visual plus inertial data processing and multi-map construction. It supports monocular, stereo and RGB-D cameras, as well as pinhole camera and fisheye camera models.

[0043] This SLAM system has added a motion segmentation module, such as... Figure 1As shown, the overall framework comprises five modules: Atlas, Tracking, Motion Segmentation, Local Mapping, Loop Closure Detection, and Map Fusion. The Atlas module consists of a series of discrete maps. It maintains a dynamic map to locate new keyframes, which is continuously updated by the Local Mapping module. Additionally, the Atlas module includes a static map. The Atlas module uses the DBoW2 keyframe database for relocalization, loop closure detection, and map fusion. The Tracking module processes sensor information, calculates the pose of the current frame and the dynamic map in real time, minimizes the matching map projection error, and determines whether the current frame is a keyframe. The Motion Segmentation module segments dynamic objects, obtains dynamic object mask images, and removes dynamic feature points before pose estimation by the Tracking module. The Local Mapping module adds keyframes and feature points to the dynamic map, removes redundant frames, and uses visual BA to optimize the map. The Loop Closure Detection and Map Fusion module detects identical regions in the dynamic map and the entire Atlas. If the identical region is in the dynamic map, a loop closure process is performed; otherwise, map fusion is performed.

[0044] 2 Motion Segmentation

[0045] 2.1 Rigidmask-based motion segmentation

[0046] Rigidmask motion estimation consists of two parts: scene object motion estimation and camera motion estimation. Scene object motion estimation uses optical flow and depth estimation methods to initialize the 3D scene flow, while camera motion estimation uses the NG-RANSAC algorithm, estimated using the epipolar geometry of two consecutive frames. Assuming the correspondence between two points of motion in two given frames is... Observations are made using a monocular camera with internal parameters (K0, K1). Based on the motion estimation results from the two parts mentioned above, the scene flow after removing the influence of camera rotation can be obtained, as shown below:

[0047]

[0048] Where K0 and K1 represent the intrinsic parameters of the monocular camera, and (Z0, Z1) are the depths corresponding to two points, estimated through object motion; R c This represents the camera's rotational motion, which is estimated through camera motion. Its homogeneous coordinates, H R =K0R c K1 -1 Used to remove camera rotation R in 2D moving scenes c The influence of this will correct the second frame image to be in the same direction as the first frame image.

[0049] After obtaining the scene stream, it is then input into a two-stream architecture. The first stream architecture uses a lightweight network, U-Net, to predict whether each pixel is part of the background. The second stream architecture uses a CenterNet network to predict the instance mask for each pixel. The second stream architecture generates coarse instance-level image masks, which refine the background mask generated by the first stream architecture. During the background and instance mask fitting process, pixels that are inconsistent between the background and instance masks are filtered out.

[0050] Rigidmask-based motion segmentation is flexible across different types of motion scenes and robust to motion noise in the observations, enabling reliable estimation and segmentation of rigid motion with only two frames. However, for some non-rigid motions, certain segmentation errors may occur. Figure 2 As shown.

[0051] 2.2 Instance Segmentation Based on YOLOv8

[0052] S01, the image is input from the input terminal, and the data is preprocessed, including operations such as blending enhancement and spatial perturbation.

[0053] S02, the preprocessed data is used for feature extraction via the backbone network Darknet.

[0054] S03, the feature maps extracted from the Backbone are passed through the Neck section to fuse feature information from feature maps of different sizes, and these features are then passed to the Head section.

[0055] S04: The Head part obtains the category, bounding box information, confidence score, and mask prototype image of each target object. Combined with these, the final instance segmentation result of the target object is obtained.

[0056] The YOLO series, as a mature object detection algorithm, features high real-time performance, high accuracy, and strong versatility. YOLOv8 improves upon previous YOLO models and adds new features.

[0057] YOLOv8's backbone network uses Darknet, a lightweight, dependency-free, and easy-to-install basic network framework. Compared to previous versions, YOLOv8 has improved its network structure by introducing more convolutional layers and residual modules to enhance the model's accuracy and robustness.

[0058] YOLOv8 employs a Feature Pyramid Network (FPN) structure. The FPN connects shallow and high-level feature maps, fusing information across different layers and predicting each feature map separately, ensuring that each feature map at different scales possesses strong semantic information. Furthermore, YOLOv8 introduces an Adaptive Anchors mechanism, which adaptively learns the target's scale and aspect ratio, improving the model's detection performance for targets of different scales and shapes. Overall, the YOLOv8 architecture integrates multiple advanced object detection techniques, improving detection accuracy and robustness while maintaining detection speed, and is widely used in real-time object detection tasks.

[0059] 2.3 Correspondence Matching

[0060] To reduce the system's error in detecting non-rigid motion, the motion segmentation module uses Rigidmask to obtain the mask image of the moving object and YOLOv8 instance segmentation to obtain the mask image. Then, the results of the two models are matched to obtain better motion segmentation results.

[0061] Assume M r and M y Rigidmask and YOLOv8 models are respectively given as inputs to a pair of original images (I i-1 ,I i ), I i-1 ,I i Two adjacent frames; Mask ri The corresponding model M r The i-th image segmentation result, and the feature points within the mask obtained from the instance segmentation are masked and labeled. Additionally, the Mask... yj Model M y The segmentation result of the j-th instance.

[0062] For the input image (I) i-1 ,I i ), through M r Get the dynamic object prediction box vertices OR i and the center point P of the rigid object c The dynamic object prediction box vertex OR i This is obtained during the mask calculation process in the Rigidmask two-stream architecture. For the center point P... c If the point is not within the object, use breadth-first search to find the nearest overmasked point P to the center point. i As the center point. Then for P iMake a judgment; if the search results are found in the Mask... yj In the middle, use Mask. yj Replace Mask ri This yields the final mask image.

[0063] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A visual SLAM method based on motion segmentation, characterized in that, Includes the following steps: S1. For the acquired RGB image, the Rigidmask motion segmentation method is used to detect potential dynamic objects and generate dynamic object mask images. The YOLO algorithm is then used for instance segmentation to obtain the object mask images. S2, perform correspondence matching on the two mask images obtained in step S1 to obtain the final mask image and obtain better motion segmentation results; The correspondence between the two mask images is matched, including: For the center point of a rigid object If the point is not within the object, use breadth-first search to find the nearest masked point to the center point. As the center point; then for Make a judgment; if the search finds... In the middle, then use replace This yields the final mask image; where It is the corresponding model The segmentation result of the i-th image. It is a model The segmentation result of the j-th instance; model and model These are Rigidmask and YOLOv8 models, respectively. S3, use the ORB algorithm to extract features from the acquired RGB image to obtain feature points; S4. Binarize the final mask image, match the feature points with the obtained binarized mask image, and determine whether they are dynamic feature points. If they are dynamic feature points, remove them. Finally, obtain the remaining static feature points for pose matching and estimation.

2. The visual SLAM method based on motion segmentation according to claim 1, characterized in that, Rigidmask, a motion segmentation method, includes: S1-1, the initial 3D scene flow is obtained by estimating the motion of scene objects, and the camera motion is obtained by estimating the motion of the camera; The motion estimation of the scene objects uses optical flow estimation and depth estimation methods; The camera's motion estimation uses the NG-RANSAC algorithm, which estimates motion based on the epipolar geometry of two consecutive frames. S1-2, based on the initial 3D scene flow and the estimation results of camera motion, the scene flow after removing the influence of camera rotation can be obtained; S1-3, input the scene stream into a dual-stream architecture to obtain the segmentation mask of rigid background and rigid motion instances; In the dual-stream architecture, stream architecture 1 is used to predict whether each pixel is the background, thus obtaining a background mask; stream architecture 2 is used to predict the instance mask for each pixel, thus obtaining an instance-level image mask; then, the instance-level image mask generated by stream architecture 2 is used to refine the background mask generated by stream architecture 1, and the background mask and instance-level image mask are fitted together to filter out pixels that are inconsistent between the background mask and the instance mask.

3. The visual SLAM method based on motion segmentation according to claim 1, characterized in that, The YOLO algorithm mentioned is YOLOv8, and YOLOv8 is used for instance segmentation.

4. A visual SLAM method based on motion segmentation according to any one of claims 1 to 3, characterized in that, The visual SLAM method is implemented based on ORB-SLAM.

5. The visual SLAM method based on motion segmentation according to claim 4, characterized in that, The ORB-SLAM is ORB-SLAM3.

Images