Visual SLAM method for indoor dynamic environment
Through optical flow method and instance segmentation network, differentiate dynamic frames and eliminate dynamic features, eliminate duplicate map points, solving the problem of traditional SLAM degradation in dynamic environments, and improving matching tracking accuracy and system robustness.
Patent Information
- Application Number
- CN202411213928.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-31
- Publication Date
- 2024-11-22
AI Technical Summary
Traditional visual SLAM algorithms have reduced positioning accuracy in dynamic environments, and existing deep learning methods have reduced tracking accuracy by eliminating feature of low dynamic targets.
The optical flow method is used to track the pose transformation between continuous frames, combine the instance to segment the network and semantic information, distinguish dynamic frames and eliminate dynamic features, eliminate duplicate map points, and improve matching tracking accuracy.
Improve the tracking accuracy and robustness of SLAM systems in indoor dynamic environments, ensuring stable tracking in high and low dynamic environments.
Smart Images

Figure CN119006526A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of visual synchronous positioning and mapping, and relates to a visual SLAM method for indoor dynamic environments. Background Art
[0002] Visual simultaneous localization and mapping (SLAM) is highly praised for its excellent mapping and positioning capabilities. However, traditional visual SLAM algorithms usually assume that the environment is static. For moving objects in dynamic scenes, the positioning accuracy will drop significantly. Many classic SLAM algorithms mainly consider static and rigid immovable objects, which poses a major challenge to posture estimation and positioning in dynamic scenes, affecting the robustness and accuracy of the algorithm. For targets in dynamic scenes, many SLAM algorithms combined with deep learning have tried to identify dynamic targets through object detection and classification in recent years, and remove their features to improve the stability of positioning. However, these methods usually remove all features on the entire target when targeting low-dynamic targets with small motion amplitudes, which in turn reduces the features that can be matched and tracked, reducing the tracking accuracy.
[0003] Therefore, a method is needed to distinguish dynamic conditions and then process dynamic features to ensure that the SLAM system can have stable tracking in high and low dynamic environments. Summary of the invention
[0004] The purpose of this method is to provide a visual SLAM method for indoor dynamic environments, aiming to solve the adaptability problem of SLAM systems in indoor dynamic environments.
[0005] To achieve the above object, the present invention provides a visual SLAM method for indoor dynamic environments, comprising:
[0006] Step 1: Input multiple consecutive frames in the image sequence into the system and use the optical flow method to track between two consecutive frames;
[0007] Step 2: After tracking using the optical flow method, the reprojection error of the matching feature point pairs of the two frames is calculated based on the pose transformation between the two consecutive frames. The reprojection error results of all matching point pairs are used to determine whether the frame is a dynamic frame.
[0008] Step 3: Based on the input of multiple consecutive frames, when the number of dynamic frames exceeds half of the number of input frames, the current frame is judged as high dynamic, otherwise it is judged as low dynamic;
[0009] Step 4: Use the instance segmentation network to perform instance segmentation on the RGB image and obtain the mask of the object in the segmented image;
[0010] Step 5: Determine whether to remove the features of dynamic targets based on the dynamics of the current frame;
[0011] Step 6: Eliminate duplicate map points in local map tracking caused by dynamic features.
[0012] A further technical solution of the present invention: the optical flow tracking in step 1 uses the LK optical flow method, which captures the motion dynamics between two consecutive frames by associating single pixels.
[0013] A further technical solution of the present invention: in the step 2, judging whether the frame is a dynamic frame according to the reprojection error results of all matching points, mainly using the RANSAC algorithm to eliminate outlier feature points, and calculating the optimal homography, and then decomposing the homography matrix to obtain the posture transformation between two consecutive frames, and calculating the reprojection error according to the posture, and then judging whether the frame is a dynamic frame according to the proportion of dynamic feature points in the total feature points;
[0014] Further technical solution of the present invention: In step 4, the image is segmented by instance, using the YOLOv8n-Seg lightweight instance segmentation network. In order to reduce the processing time, TensorRT, a GPU acceleration method, is used in the program to speed up the processing speed of the program.
[0015] A further technical solution of the present invention: in the step 5, whether to remove the features of the dynamic target is determined by the dynamics of the current frame, and the features existing in the dynamic target are removed. A mask containing only the dynamic target is retained mainly based on semantic information, and the mask is used to remove only the features on the dynamic target.
[0016] A further technical solution of the present invention is as follows: in step 6, repetitive map points in local map tracking are eliminated, mainly by using the DBow2 bag-of-words model to accelerate the matching of map point descriptors belonging to the same Nodeld current frame and map point descriptors in local map tracking, and after accelerated matching, map points with high similarity between the current frame and the local map tracking are searched, and these map points are deleted.
[0017] In summary, the present invention discloses a visual SLAM method for indoor dynamic environments, including: inputting multiple consecutive frames into a system in an image sequence, and using an optical flow method to track between two consecutive frames; after tracking using the optical flow method, calculating the reprojection error of the matching feature point pairs of the two frames by the posture transformation between the two consecutive frames, and judging whether the frame is a dynamic frame according to the reprojection error results of all matching point pairs; according to the input of multiple consecutive frames, when the number of dynamic frames exceeds half of the number of input frames, the current frame is judged to be highly dynamic, and vice versa; using an instance segmentation network to perform instance segmentation on the RGB image, and obtaining a mask after the object in the segmented image; determining whether to remove the features of the dynamic target according to the dynamics of the current frame; and eliminating repeated map points in local map tracking caused by dynamic features. Compared with many existing SLAM systems combined with deep learning, the present invention uses a dynamic state algorithm that can distinguish dynamic targets in indoor dynamic environments and uses instance segmentation to perform instance segmentation on images, and then combines semantic information to retain the mask of dynamic targets that may move, effectively solving the problem of retaining the features of low-dynamic targets to enable matching tracking to have more matching features, and eliminating the features of high-dynamic targets, effectively preventing the problem of dynamic features causing a serious decrease in matching tracking accuracy. In addition, the added duplicate map point elimination algorithm should eliminate duplicate map points under dynamic feature retention, further improving the tracking accuracy of the SLAM system. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a schematic diagram of the process of the present invention.
[0019] Figure 2 It is a flow chart for calculating the inter-frame reprojection error.
[0020] Figure 3 This is the instance segmentation result using YOLOv8n-Seg. (a) is the original image, (b) is the instance segmentation result, and (c) is the mask obtained after segmentation.
[0021] Figure 4 It is a mask that only retains dynamic targets that may move.
[0022] Figure 5 It is a flow chart of dynamic feature elimination.
[0023] Figure 6 It is a flowchart for eliminating duplicate map points DETAILED DESCRIPTION
[0024] Embodiments of the method are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the method, but should not be understood as limiting the method.
[0025] See also Figure 1 The present invention provides a visual SLAM method for indoor dynamic environments, comprising the following steps:
[0026] Step 1: Input multiple consecutive frames in the image sequence into the system and use the optical flow method to track between two consecutive frames;
[0027] Step 2: After tracking using the optical flow method, the reprojection error of the matching feature point pairs of the two frames is calculated based on the pose transformation between the two consecutive frames. The reprojection error results of all matching point pairs are used to determine whether the frame is a dynamic frame:
[0028] Specific method:
[0029] like Figure 2 As shown in the figure, after the optical flow method tracks the feature points of the previous frame, the RANSAC (random sampling consensus) algorithm is used to eliminate outlier feature points and calculate the optimal homography. The inter-frame pose transformation matrix T is then obtained by decomposing the homography matrix. j,j-1 ∈SE(2), and finally use the pose transformation matrix to make the reprojection error of the two frame matching feature point pairs. The process is as follows Figure 3 The re-projection error formula is as follows:
[0030] e i =||T j,j-1 u j-1,i -u j,i ||
[0031] Among them, u j,i Indicates the pixel position of the feature point in the current image frame, e i represents the reprojection error, and j represents the current frame. Whether the feature point in each frame is a dynamic point is determined by whether the reprojection error is greater than the threshold as follows:
[0032]
[0033] Where a represents a threshold value, which is set to 1.7 pixels. When the reprojection error is greater than a, it means that the feature point is a dynamic point. i A flag indicating whether the feature point is a dynamic point.
[0034] When the percentage of dynamic points in a frame to the total feature points of the frame reaches a certain threshold, the frame can be judged as dynamic, as shown in the following formula:
[0035]
[0036] Where b indicates that the dynamic threshold is set to 10%, W j Represents all feature points of the frame, HF j Indicates whether the frame is dynamic;
[0037] Step 3: Based on the input of multiple consecutive frames, when the number of dynamic frames exceeds half of the number of input frames, the current frame is judged as high dynamic, otherwise it is judged as low dynamic:
[0038] Specifically, HF is obtained from step 2 j , input multiple consecutive frames, when these frames belong to HF j When the number of frames exceeds half of the number of input frames, the current frame is considered dynamic, and the number of consecutive input frames is 10. Because multiple frames need to be input, the high and low dynamic threads must be initialized at the beginning of the program. The first 10 frames are judged as all high dynamic or low dynamic according to the 10th frame;
[0039] Step 4: Use the instance segmentation network to perform instance segmentation on the RGB image and obtain the mask after segmenting the objects in the image:
[0040] Specifically, Figure 3 As shown in the figure, instance segmentation is performed, and the lightweight model YOLOv8n-Seg is used to detect and segment RGB images. In order to reduce processing time, TensorRT, a GPU acceleration method, is used in the program to speed up the processing speed of the program. Then, according to the semantic information, the mask containing only dynamic targets is retained, as shown in the figure. Figure 4 As shown;
[0041] Step 5: Determine whether to remove the features of dynamic targets based on the current frame dynamics:
[0042] Specifically, Figure 5 As shown in the figure, the result of step 3 is used to determine whether the current frame is high dynamic, and then the instance segmentation result is used to remove the feature points on the character. In this way, when the character is in a low dynamic situation, the feature points on the character will not be removed. It is guaranteed that there are enough feature points for subsequent matching and tracking. Compared with completely removing the feature points on the character in a low dynamic situation, it is more stable to distinguish the dynamic situation and then process and track.
[0043] Step 6: Eliminate duplicate map points in local map tracking caused by dynamic features:
[0044] Specifically, Figure 6As shown in the figure, DBoW2 is used to accelerate the matching of the current map point and the local map point. When the current frame map point descriptor and the local map point descriptor belong to the same node (Nodeld), the map point feature descriptor of the same Nodeld is found according to the index of the feature vector, and the distance between the current frame map point descriptor and the local map point descriptor is calculated, that is, their Hamming correlation distance is calculated. The distance calculation formula is as follows: dist(cd,ld k )=||cd-ld k ||
[0045] Among them, cd represents the current frame point descriptor, ld k Represents the local map point descriptor. The distance formula can be used to calculate the distance between the descriptor of each map point in the current frame and the local map point descriptor. Based on these distance sets, distances greater than 0 and less than the median of the distance set are removed, which can eliminate repetitive map points caused by dynamic features.
Claims
1. A visual SLAM method for indoor dynamic environments, characterized in that: The following steps are involved: Step 1: Input multiple consecutive frames in the image sequence into the system and use the optical flow method to track between two consecutive frames; Step 2: After tracking using the optical flow method, the reprojection error of the matching feature point pairs of the two frames is calculated based on the pose transformation between the two consecutive frames. The reprojection error results of all matching point pairs are used to determine whether the frame is a dynamic frame. Step 3: Based on the input of multiple consecutive frames, when the number of dynamic frames exceeds half of the number of input frames, the current frame is judged as high dynamic, otherwise it is judged as low dynamic; Step 4: Use the instance segmentation network to perform instance segmentation on the RGB image and obtain the mask of the object in the segmented image; Step 5: Determine whether to remove the features of dynamic targets based on the dynamics of the current frame; Step 6: Eliminate duplicate map points in local map tracking caused by dynamic features.
2. A visual SLAM method for indoor dynamic environments as claimed in claim 1, characterized in that The optical flow tracking in step 1 uses the LK optical flow method, which captures the motion dynamics between two consecutive frames by associating single pixels.
3. A visual SLAM method for indoor dynamic environments as claimed in claim 1, characterized in that In step 2, whether the frame is a dynamic frame is determined based on the reprojection error results of all matching points. The RANSAC algorithm is mainly used to eliminate outlier feature points, and the optimal homography is calculated. The homography matrix is then decomposed to obtain the posture transformation between two consecutive frames. The reprojection error is calculated based on the posture, and then the proportion of dynamic feature points in the total feature points is used to determine whether the frame is a dynamic frame.
4. A visual SLAM method for indoor dynamic environments as claimed in claim 1, characterized in that In step 4, the image is segmented using the YOLOv8n-Seg lightweight instance segmentation network. In order to reduce processing time, TensorRT, a GPU acceleration method, is used in the program to speed up the processing speed of the program.
5. A visual SLAM method for indoor dynamic environments as claimed in claim 1, characterized in that In step 5, the dynamics of the current frame determines whether to remove the features of the dynamic target. The features existing in the dynamic target are removed, and a mask containing only the dynamic target is retained mainly based on semantic information. The mask is used to remove only the features on the dynamic target.
6. A visual SLAM method for indoor dynamic environments as claimed in claim 1, characterized in that In step 6, repeated map points in the local map tracking are eliminated mainly by using the DBow2 bag-of-words model to accelerate the matching of the map point descriptors in the local map tracking of the same Nodeld with the map point descriptors of the current frame, and after accelerated matching, map points with high similarity between the current frame and the local map tracking are found and deleted.
Citation Information
Cited By
Dynamic SLAM (Simultaneous Localization and Mapping) pose resolving method and system
CN121616646A
