A robust visual SLAM method based on deep learning in dynamic scenes

Abstract

The present invention discloses a robust visual SLAM method based on deep learning in a dynamic scene, belonging to the field of artificial intelligence, robotics and computer vision. The present invention adopts a camera as an image acquisition device. Firstly, the image sequence acquired by the camera is divided into static objects and dynamic objects using the Mask R-CNN semantic segmentation network based on deep learning. The pixel-level semantic segmentation of the dynamic object is used as the semantic prior knowledge and the feature points on the dynamic object are eliminated; the geometric constraint of the epipolar geometric characteristics is used to further check whether the feature is a dynamic feature; and a complete robust visual SLAM system is formed by combining the local mapping and loop detection modules. The present invention can well reduce the absolute trajectory error and relative pose error of the SLAM system, and improve the accuracy and robustness of the pose estimation of the SLAM system.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, robotics and computer vision, and in particular relates to a robust visual SLAM method based on deep learning in dynamic scenes. Background Art

[0002] In recent years, simultaneous localization and mapping (SLAM) has become an important research field in artificial intelligence, robotics and computer vision. Localization and mapping in dynamic scenes is one of the hottest research directions and is widely used in indoor service robots, outdoor self-driving cars and other fields.

[0003] Currently, most visual SLAM methods are based on the assumption that the observed environment is static. Since the real environment contains dynamic objects, traditional SLAM methods can easily lead to insufficient feature matching due to incorrect correspondence or occlusion of tracking features, resulting in drift or even loss of pose estimation, resulting in low accuracy and poor robustness of the system in dynamic environments.

[0004] Some existing methods for detecting dynamic features rely on geometric information. By defining geometric constraints, if the matching features in subsequent frames are far away from the polar lines, they are considered dynamic, or the features of the previous frame are projected to the current frame, and the reprojection error of feature tracking is calculated. If the error is large, it is considered dynamic. These methods rely solely on geometric information and cannot establish prior semantic information of dynamic objects, and cannot provide a high-level understanding of dynamic objects in the surrounding environment, resulting in poor accuracy of the system in dynamic environments. Some methods rely on semantic information. Based on human experience and attempts, objects that can move by themselves are considered dynamic objects. The semantic information of dynamic objects is obtained through target detection or semantic segmentation, and the feature points on dynamic objects are removed. These methods cannot effectively remove unstable feature points on potential dynamic objects.

[0005] Dynamic feature detection is a key step in building a robust visual SLAM. Only when dynamic features are well detected can they be eliminated and pose estimation can be performed using stable static feature points. Due to the complexity of dynamic scenes and the influence of factors such as incorrect correspondence or occlusion of tracking features, detecting dynamic features in dynamic scenes to build a robust visual SLAM has become a very challenging task. An advanced deep learning method can be used to detect dynamic objects, and a semantic segmentation network can be used to segment dynamic objects. Pixel-level semantic segmentation of dynamic objects can be obtained as semantic prior knowledge, and a motion consistency detection algorithm can be used to further detect potential dynamic moving objects. The dynamic features can be better detected by combining semantic information with geometric information. Summary of the invention

[0006] The object of the present invention is to overcome the above-mentioned shortcomings of the prior art, and adopt a robust visual SLAM method based on deep learning in dynamic scenes. The method combines semantic segmentation technology with a motion consistency detection algorithm. First, the MaskR-CNN network is used to perform semantic segmentation on the image, and the prior semantic information of the dynamic object is established. Then, the feature points belonging to the dynamic object are further eliminated through the motion consistency detection algorithm, and finally, the static feature points are used for feature matching and pose estimation. The present invention can reduce the absolute trajectory error and relative pose error of the SLAM system, and improve the accuracy and robustness of the pose estimation of the SLAM system.

[0007] The technical solution adopted by the present invention is a robust visual SLAM method based on deep learning in dynamic scenes, which comprises the following steps:

[0008] (1) Using a deep learning-based semantic segmentation network to divide the objects in the image sequence captured by the camera into static objects and dynamic objects, the pixel-level semantic segmentation of dynamic objects is used as semantic prior knowledge;

[0009] 1.1) Build the Mask R-CNN segmentation network and use the Mask R-CNN network to perform semantic segmentation of potential dynamic objects. In order to segment dynamic objects and cover the mask, first set the Mask R-CNN network model and its loss function. Mask R-CNN will complete the three tasks of mask prediction, regression prediction and classification prediction. The loss function uses the average binary cross entropy loss and the binary cross entropy loss. The overall loss function of the Mask R-CNN network training is the sum of the classification error, regression error and segmentation error:

[0010] L=L cls +L box +L mask

[0011] Where L is the overall loss function of network training; L mask is the loss function for mask prediction; L cls is the loss function for classification prediction; L box is the loss function for regression prediction.

[0012] 1.2) Use the MS COCO dataset as training samples to train the Mask R-CNN network. The MS COCO dataset contains more than 80 different object categories. The required categories are divided into two types: one is moving objects with high dynamic confidence, such as people, cars, cats, dogs, etc.; the other is objects with high static confidence or potentially dynamic moving objects, such as chairs, books, cups, etc.

[0013] 1.3) The input of the Mask R-CNN network is an m×n×3 RGB image, and the output is an m×n×l matrix, where l is the number of objects in the image. For each output channel i∈l, a binary mask is obtained, with only two numbers, 1 and 0, in the matrix. Number 1 corresponds to static scenes, and number 0 corresponds to dynamic objects. By merging all output channels into one channel, the segmentation of all dynamic objects appearing in a scene image can be obtained.

[0014] (2) Since the semantic segmentation results near the object boundary are still fuzzy and cannot effectively remove unstable feature points on potential dynamic objects, it is necessary to use the geometric constraints of the epipolar geometry to further check whether the feature is a dynamic feature. The specific steps are as follows:

[0015] 2.1) Extract ORB feature points from the input image sequence;

[0016] 2.2) Use the optical flow method to obtain the movement of pixels in the image, so as to achieve pixel tracking; the specific steps are as follows:

[0017] 2.2.1) Assume that the image captured by the camera changes with time, and the grayscale value of the pixel (x, y) in the image is I(x, y, t). Since the grayscale value of the same spatial point remains consistent on the plane as time changes, it satisfies: I(x, y, t) = I(x+dx, y+dy, t+dt)

[0018] Assuming that the motion is relatively small (i.e., the change in time does not cause a drastic change in the target position, and the displacement between adjacent frames reaches the millimeter level), the expansion is:

[0019]

[0020] Combining the two equations, we get:

[0021]

[0022] is the speed of the point along the x-axis, is the speed of the point along the y-axis. The optical flow algorithm assumes that the pixel movement in the image block is the same. Finally, through multiple iterations, the movement of the pixel in the image can be obtained, thereby realizing the tracking of the pixel point.

[0023] 2.2.2) Based on the feature point set P1 of the previous frame, the optical flow method is used to calculate the matching feature point set P2 in the current frame. If the matching pair is too close to the edge of the image, or the pixel difference of the 3×3 image block in the center of the matching pair is too large, the matching pair will be discarded.

[0024] 2.3) Estimate the fundamental matrix F using the classic eight-point method;

[0025] 2.4) Assume that p1 and p2 are a pair of matching feature points, and their homogeneous coordinates are as follows:

[0026] P1=[u1,v1,1],P2=[u2,v2,1]

[0027] Where u and v are the corresponding pixel horizontal and vertical coordinates, and the basic matrix F and feature points are used to calculate the epipolar lines corresponding to the feature points in the current frame:

[0028]

[0029] 2.5) Calculate the relationship between the distance from p2 to the corresponding epipolar line of p1 and the predetermined threshold to determine whether the feature point has moved. The distance D between point p2 and the epipolar line l1 is calculated by the following formula:

[0030]

[0031] If the distance is greater than a predetermined threshold, p2 is added to the dynamic feature point set S; otherwise, it is regarded as a relatively stable static feature point for subsequent pose estimation.

[0032] (3) In the previous tracking thread, a new keyframe is obtained. Local mapping is performed by inserting keyframes, deleting redundant map points and keyframes, and adjusting local clusters. The specific steps are as follows:

[0033] 3.1) Add the new keyframe as a new node to the graph and update the edges connected to the keyframe nodes that can share map points. At the same time, update the growth tree of the keyframe and calculate the bag of words representing the keyframe.

[0034] 3.2) To save the map point, the constraint must be tested in the first three frames of the point cloud. If the point can be observed in less than three key frames, the current map point will be deleted;

[0035] 3.3) For unmatched points, match them with unmatched feature points in other keyframes. After triangulation of ORB feature point pairs, check forward depth of field, parallax, back-projection error and scale consistency, and create new map points.

[0036] 3.4) Local bundle adjustment optimizes the currently processed key frame;

[0037] 3.5) To control the compactness of reconstruction, redundant keyframes are detected through local mapping. Keyframes whose points can be observed by more than three keyframes are considered to be redundant keyframes and deleted.

[0038] (4) Since the visual odometer will have cumulative errors during the tracking process, a closed loop is used to detect whether the camera has been here before, and back-end optimization is performed to eliminate the cumulative errors caused by the camera movement during this period. The specific steps are as follows:

[0039] 4.1) Check whether there is a new key frame in the sequence. If the number of key frames in the map is less than 10, no closed loop detection is performed; 4.2) Detect closed loop:

[0040] 4.2.1) Get the common view keyframes and calculate the BoW score between them and the current keyframe to get the lowest score;

[0041] 4.2.2) Find the candidate key frames in the database by calculating the lowest score, which is equivalent to finding the key frames that have been here;

[0042] 4.2.3) Perform continuity check on the candidate key frame set.

[0043] 4.3) The closed loop has been detected in the previous step, and backend optimization is required. The main work is to find more corresponding points between the current key frame and the closed loop frame, and calculate the Sim3 transformation between the current key frame and the closed loop frame through these corresponding points to solve Rt and s;

[0044] 4.4) Calculate the closed loop and fused bitmap. Finally, more effective data fusion of all matching point clouds is achieved.

[0045] The present invention adopts a camera as an image acquisition device, and both a binocular camera and a depth camera may be used.

[0046] Compared with the existing SLAM technology based on static environment assumption, the present invention can reduce the absolute trajectory error and relative pose error of the SLAM system by combining the Mask R-CNN semantic segmentation network based on deep learning and the motion consistency detection algorithm, thereby improving the accuracy and robustness of the SLAM system pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Flow chart of the method of the present invention.

[0048] Figure 2 This is the flow chart of motion consistency detection algorithm. DETAILED DESCRIPTION

[0049] Figure 1 is a flow chart of the method of the present invention, referring to Figure 1The present invention provides a robust visual SLAM method based on deep learning in dynamic scenes. Four threads run in parallel in the system: tracking, semantic segmentation, local map, and closed-loop detection. When the original RGB image arrives, it is simultaneously passed to the semantic segmentation thread and the tracking thread, and the two process the image in parallel. The semantic segmentation thread uses the Mask R-CNN network to divide objects into dynamic objects and static objects, provides the pixel-level semantic labels of dynamic objects to the tracking thread, and further detects potential dynamic feature point outliers through the motion consistency detection algorithm of geometric constraints, and then removes the ORB feature points in the dynamic objects and uses relatively stable static feature points for pose estimation. And by inserting key frames, deleting redundant map points and key frames, and performing local mapping through local cluster adjustment, as well as detecting whether you have been here before through closed loops, back-end optimization is performed to reduce the cumulative error to an acceptable range.

[0050] Figure 2 For the motion consistency detection algorithm flow chart, refer to Figure 2 , in order to detect the dynamic features of potentially dynamic moving objects, firstly, based on the feature point set P1 of the previous frame, the optical flow method is used to calculate the matching feature point set P2 in the current frame. If the matching pair is too close to the edge of the image, or the pixel difference of the 3×3 image block in the center of the matching pair is too large, the matching pair will be discarded. Then, at least five pairs of features can be used to estimate the basic matrix F, usually using the classic eight-point method, and then the basic matrix F is used to calculate the epipolar lines in the current frame. Finally, the relationship between the distance from p2 to the corresponding epipolar line of p1 and the predetermined threshold is calculated to determine whether the feature point has moved.

[0051] When the robot's perceived environment contains dynamic objects, it is necessary to detect dynamic features from adjacent frames using a motion consistency detection algorithm.

[0052] The above description is only an embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A robust visual SLAM method based on deep learning in dynamic scenes, characterized by: The method is implemented based on a visual sensor camera and specifically includes the following steps: S1 uses a deep learning-based semantic segmentation network to divide the objects in the image sequence captured by the camera into static objects and dynamic objects, and uses the pixel-level semantic segmentation of dynamic objects as semantic prior knowledge; S1.1 Build the Mask R-CNN segmentation network and use the Mask R-CNN network to perform semantic segmentation of potential dynamic objects; S1.2 uses the MS COCO dataset as training samples to train the Mask R-CNN network; S1.3 The input of the Mask R-CNN network is an m×n×3 RGB image, and the output is an m×n×l matrix. By merging all output channels into one channel, the segmentation of all dynamic objects appearing in a scene image is obtained; S2 uses the motion consistency detection algorithm to check whether the feature is a dynamic feature; S2.1 extracts ORB feature points from the input image sequence; S2.2 uses the optical flow method to obtain the movement of pixels in the image, thereby achieving pixel tracking; S2.3 estimates the fundamental matrix F using the classic eight-point method; S2.4 uses the basic matrix F and the feature points to calculate the epipolar lines corresponding to the feature points in the current frame; S2.5 calculates the relationship between the distance of the polar line and the predetermined threshold to determine whether the feature point moves; S3 performs local mapping by inserting keyframes, deleting redundant map points and keyframes, and adjusting local clusters. The steps are as follows: 3.1) Add the new keyframe as a new node to the graph, and update the edges connected to the keyframe nodes that can share map points; at the same time, update the growth tree of the keyframe and calculate the bag of words representing the keyframe. 3.2) To save the map point, the constraint must be tested in the first three frames before the node is created. If the node can be observed in less than three key frames, the current map point will be deleted; 3.3) For unmatched points, match them with unmatched feature points in other keyframes. After triangulation of ORB feature point pairs, check forward depth of field, parallax, back-projection error and scale consistency, and create new map points. 3.4) Local bundle adjustment optimizes the currently processed key frame; 3.5) To control the compactness of reconstruction, redundant keyframes are detected through local mapping. Keyframes whose points can be observed by more than three keyframes are considered redundant keyframes and deleted. S4 uses a closed loop to detect whether the user has been here before, performs backend optimization, and reduces the accumulated error to a certain range. The steps are as follows: 4.1) Check whether there are new key frames in the sequence. If the number of key frames in the map is less than 10, no closed loop detection is performed; 4.2) Detection closed loop: 4.2.1) Get the common view keyframes and calculate the BoW score between them and the current keyframe to get the lowest score; 4.2.2) Use the calculated minimum score to find candidate key frames in the database and find the key frames that have been here; 4.2.3) Perform continuity detection on the candidate key frame set; 4.3) The closed loop has been detected in the previous step, and backend optimization is required; find more corresponding points between the current key frame and the closed loop frame, and calculate the Sim3 transformation between the current key frame and the closed loop frame through these corresponding points to solve Rt and s; 4.4) Calculate the closed loop and fuse the bitmap; ultimately achieve more effective data fusion of all matching point clouds.

2. The robust visual SLAM method based on deep learning in a dynamic scene according to claim 1, characterized in that: The robust visual SLAM system utilizes steps S1 to S4 to be applied to the required scenes with dynamic objects.

Images