Real-time semantic vSLAM algorithm based on depth map inpainting

Abstract

The present invention discloses a real-time semantic vSLAM algorithm based on depth map repair, and relates to the field of computer vision technology. The real-time semantic vSLAM algorithm based on depth map repair adopts a non-blocking tracking thread structure, the semantic thread segments the latest input frame, and the tracking thread uses the latest segmented frame, so that the tracking thread does not have to wait for the semantic thread segmentation to be completed before starting the pose calculation, thereby improving the real-time performance of the system and meeting the application of the real-world SLAM system. The Deeplabv3+ semantic segmentation algorithm is used to obtain the semantic segmentation map of the image, which is faster than the segmentation speed of Segnet and Mask‑RCNN used in the prior art, and can also meet the requirements of high-precision segmentation results. The LK multi-layer optical flow method is used to pair ORB key points to eliminate dynamic points and make up for semantic delays. It is also proposed to use a depth map to restore a semantic image method to repair the missing parts of the semantic segmentation and improve the segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a real-time semantic vSLAM algorithm based on depth map restoration. Background Art

[0002] Visual simultaneous localization and mapping (vSLAM) refers to a technology that uses a camera mounted on a robot to sense the surrounding environment, build a model of the environment, and estimate its own movement during movement without prior information about the environment. It is widely used in driverless cars, AGV handling robots, drones, and other fields. vSLAM usually has a rigid scene assumption, which assumes that the camera is the only moving object in the environment. However, in the densely populated real world, the camera is not the only moving object. This assumption can easily lead to system tracking and positioning failures, which greatly interferes with the positioning and mapping of the vSLAM system, making it impossible for the robot to accurately locate itself in a complex environment. Therefore, the study of SLAM algorithms in dynamic environments has become important.

[0003] In the prior art, with the rapid development and widespread application of artificial neural networks, many SLAM solutions for complex environments have tried to use object detection or semantic segmentation to eliminate the impact of dynamic objects on the system. DynaSLAM uses the Mask R-CNN instance segmentation network to detect dynamic objects and combines multiple views to eliminate dynamic points. DS-SLAM uses the semantic segmentation network SegNet to detect dynamic objects and uses motion consistency detection to assist in eliminating system dynamic points. RDS-SLAM is an improved real-time semantic SLAM system based on ORB-SLAM3. Its semantic segmentation thread and tracking thread are separated. During operation, semantic segmentation is only performed in key frames, and the motion probability of feature points is updated according to the segmentation results. It analyzes and compares the semantic delay of various key frame selection schemes in detail, and gives a key frame selection strategy that is suitable for segmentation networks of various rates, which can minimize semantic delay and tap the potential of semantic information.

[0004] However, the tracking threads of DynaSLAM and DS-SLAM need to wait for the semantic segmentation thread to end before estimating the pose of the next frame. Usually, the segmentation speed is slow, which will inhibit the processing speed of the robot's vSLAM in a real-time environment, making the robot unable to respond quickly and reducing the real-time performance of the system. The RDS-SLAM system determines whether it is a dynamic point by updating the movement probability of key points. However, this system only uses key frames and ignores the semantic information of most ordinary frames. The semantic information provided is not much, and the system positioning accuracy is not high enough. These semantic SLAM systems have a common disadvantage, because the system will cause inaccurate semantic segmentation under uncertain factors such as the camera moving too fast, which will reduce the accuracy of the system.

[0005] In view of this, we propose a real-time semantic vSLAM algorithm based on depth map inpainting. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] In view of the deficiencies in the prior art, the present invention provides a real-time semantic vSLAM algorithm based on depth map restoration, which solves the problems mentioned in the above background technology.

[0008] (II) Technical solution

[0009] To achieve the above objectives, the present invention is implemented by the following technical scheme: a real-time semantic vSLAM algorithm based on depth map repair, the real-time visual SLAM algorithm comprises the following steps:

[0010] S1. Input the RGB-D image frame taken by the depth camera, and pass the RGB-D image to the semantic segmentation thread and tracking thread respectively, and update the image to be segmented;

[0011] S2, adopts a structure that does not block the tracking thread. The semantic thread uses DeeplabV3+ based on the MobileNetV2 framework to segment the latest image to be segmented and obtain the latest semantic segmentation map;

[0012] S3, repairing the segmentation map, merging the segmentation map and the corresponding deep restored semantic image as the latest semantic segmentation frame;

[0013] S4, convert the RGB image of the current frame into a grayscale image, and extract its ORB key points;

[0014] S5, the tracking thread uses the latest semantic segmentation map without waiting for the semantic segmentation thread to end, and uses the LK multi-layer optical flow method to remove the dynamic key points of the current frame;

[0015] S6, restore the semantic image using ORB pairing points and depth information;

[0016] S7, using the remaining static key points to estimate or relocate the camera's pose;

[0017] S8. In dense mapping, semantic information and tracking information are used to build a semantic point cloud map.

[0018] Optionally, S2 includes:

[0019] S21. Use libtorch to call the pt file of DeeplabV3+ and load the DeeplabV3+ semantic segmentation network model based on the MobileNetV2 framework.

[0020] S22, when there is a latest image input, update the latest frame to be segmented;

[0021] S23, after the semantic segmentation thread completes the segmentation of the previous image, it segments the latest image to be segmented.

[0022] Optionally, the S5 includes:

[0023] S51, obtaining the ORB key points and grayscale images of the current frame and the latest segmented frame;

[0024] S52, set the latest segmentation frame (x 1 ,y 1 ) is grayscale I 1 (x 1 ,y 1 ), current frame (x 2 ,y 2 ) is grayscale I 2 (x 2 ,y 2 ), based on the assumption that the grayscale is constant, the grayscale of the same point is constant, and the Gauss-Newton formula is used to track the movement of ORB key points. The Gauss-Newton formula is

[0025] S53, using the original image as the bottom layer of the image pyramid, scaling the lower layer image by a certain factor each time going up a layer, to obtain images of different resolutions, and when calculating the optical flow tracking ORB key points, starting with the top layer image, and then using the grayscale calculation result of the previous layer as the initial value of the next layer to calculate, until the calculation reaches the bottom layer, completing the pairing of the ORB key points;

[0026] S54, determining whether the semantic label of the paired point of the current image ORB key point on the latest segmented frame RGB image on its semantic mask is a dynamic object, and if so, marking the ORB key point as a dynamic key point and then removing it.

[0027] Optionally, S6 includes:

[0028] S61. Determine whether the current frame ORB key point is a dynamic point and whether it has not been traversed. If so, perform a depth-first traversal using the breadth-first algorithm;

[0029] S62. Create a new queue a, put the ORB key point at the end of the queue, and obtain the depth value of this point as the base depth d for traversal.

[0030] S63. Initialize the queue: head = 0, tail = 1;

[0031] S64. When head!= tail, increment head, and start traversing in the up, down, left, and right directions of the point corresponding to a[head], and record its depth value as d n (n = 1, 2, 3, 4);

[0032] S65. If d - 0.1 < d n < d + 0.1, then mark this point as a dynamic point, denoted as 1, increment tail, and put it at the end of the queue to be traversed. Otherwise, it is a static point, denoted as 0, and continue to search the next direction.

[0033] S66. Repeat the above steps 6.2 to 6.5 until all points in the traversal queue have been traversed, and then perform the traversal of the next ORB key point;

[0034] S67. When all ORB key points have been traversed, obtain the depth-restored semantic image and pass it into the semantic segmentation thread for the semantic thread to repair the segmentation map.

[0035] Optionally, S8 includes:

[0036] S81. Obtain the key frame pose information, RGB-D image information, and its semantic segmentation map passed in by the tracking thread;

[0037] S82. Obtain each pixel P on the RGB image uv , if the semantic label of this pixel is a person, then do not add it to the point cloud map;

[0038] S83. Each pixel point P uv has the following transformation relationship with the map point, where T cw is the transformation matrix from the world coordinate system to the camera coordinate system, K is the camera internal parameter matrix, and Z is the depth value of the pixel point P uv ;

[0039]

[0040] S84. Use the following formula to transform the two-dimensional pixel point P uvProject into three-dimensional space to get map point P w The coordinates of

[0041]

[0042] S85. Output the point cloud map.

[0043] (III) Beneficial effects

[0044] The present invention provides a real-time semantic vSLAM algorithm based on depth map repair, which has the following beneficial effects:

[0045] (1) The real-time semantic vSLAM algorithm based on depth map restoration adopts a non-blocking tracking thread structure. The semantic thread segments the latest input frame, and the tracking thread uses the latest segmented frame. This makes it unnecessary for the tracking thread to wait for the semantic thread segmentation to be completed before starting pose calculation, thereby improving the real-time performance of the system and meeting the application requirements of real-world SLAM systems.

[0046] (2) This real-time semantic vSLAM algorithm based on depth map restoration uses the Deeplabv3+ semantic segmentation algorithm to obtain the semantic segmentation map of the image. It is faster than the Segnet and Mask-RCNN used in the existing technology and can also meet the requirements of high-precision segmentation results. The LK multi-layer optical flow method is used to pair ORB key points to eliminate dynamic points and compensate for semantic delay.

[0047] (3) The real-time semantic vSLAM algorithm based on depth map repair uses the depth restored semantic image to repair the missing parts of semantic segmentation. The system directly uses the images obtained by semantic segmentation, and segmentation is prone to segmentation loss in some scenes such as blur and rotation, resulting in reduced system accuracy. The depth restored semantic image method can use the LK optical flow to pair with the ORB key points of the current frame to restore the semantic map. Because the system uses a non-blocking thread structure, there is a certain delay when the semantic information is compared with the current frame. This method is completed when the tracking thread removes dynamic points. Therefore, after the semantic segmentation is completed, the semantic thread can always obtain the depth restored semantic image of the picture to repair the semantic segmentation map, and use future semantic information to repair the semantic segmentation map. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the overall process structure of the present invention;

[0049] Figure 2 This is a flowchart of deep semantic image restoration of the present invention; DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0052] See also Figure 1 The present invention provides a technical solution: a real-time semantic vSLAM algorithm based on depth map repair, the real-time visual SLAM algorithm comprises the following steps:

[0053] S1. Input the RGB-D image frame taken by the depth camera, and pass the RGB-D image to the semantic segmentation thread and tracking thread respectively to update the image to be segmented.

[0054] S2, adopts a structure that does not block the tracking thread. The semantic thread uses DeeplabV3+ based on the MobileNetV2 framework to segment the latest image to be segmented and obtain the latest semantic segmentation map, which further includes:

[0055] S21. Use libtorch to call the pt file of DeeplabV3+ and load the DeeplabV3+ semantic segmentation network model based on the MobileNetV2 framework.

[0056] S22: When the latest image is input, the latest frame to be segmented is updated.

[0057] S23, after the semantic segmentation thread completes the segmentation of the previous image, it segments the latest image to be segmented;

[0058] S3. Repair the segmentation map and merge the segmentation map and the corresponding deep restored semantic image as the latest semantic segmentation.

[0059] S4. Convert the RGB image of the current frame into a grayscale image and extract its ORB key points.

[0060] S5. The tracking thread uses the latest semantic segmentation map without waiting for the semantic segmentation thread to end, and uses the LK multi-layer optical flow method to remove the dynamic key points of the current frame, further including:

[0061] S51, obtaining the ORB key points and grayscale images of the current frame and the latest segmented frame.

[0062] S52, set the latest segmentation frame (x 1 ,y 1 ) is grayscale I 1 (x 1 ,y 1 ), current frame (x 2 ,y 2 ) is grayscale I 2 (x 2 ,y 2 ), based on the assumption that the grayscale is constant, the grayscale of the same point is constant, and the Gauss-Newton formula is used to track the movement of ORB key points. The Gauss-Newton formula is

[0063] S53, using the original image as the bottom layer of the image pyramid, scaling the lower layer image by a certain factor each time going up a layer, to obtain images of different resolutions. When calculating the optical flow tracking ORB key points, the calculation starts from the top layer image first, and then the grayscale calculation result of the previous layer is used as the initial value of the next layer to calculate, until the calculation reaches the bottom layer, completing the pairing of the ORB key points.

[0064] S54, determine whether the semantic label of the paired point of the current image ORB key point on the latest segmented frame RGB image on its semantic mask is a dynamic object (here only the dynamic object of human is considered), if so, mark the ORB key point as a dynamic key point and then remove it.

[0065] S6, using ORB pairing points and depth information to restore semantic images (steps such as Figure 2 ), further comprising:

[0066] S61, determine whether the ORB key point of the current frame is a dynamic point and whether it has not been traversed. If so, perform a depth traversal using a breadth-first algorithm.

[0067] S62: Create a new queue a, put the ORB key point at the end of the queue, and obtain the depth value of the point as the basic depth d of the traversal.

[0068] S63. Initialize the queue head=0, tail=1.

[0069] S64, when head! = tail, head++, start traversing in the up, down, left, and right directions of the point corresponding to a[head], and record its depth value as d n (n=1,2,3,4).

[0070] S65, if d-0.1 <d nIf d + 0.1, then mark this point as a dynamic point, denoted as 1, tail++, and put it at the end of the queue to be traversed. Otherwise, it is a static point, denoted as 0, and continue to search the next direction.

[0071] S66. Repeat the above steps from 6.2 to 6.5 until all points in the traversal queue have been traversed, and then perform the next ORB key point traversal.

[0072] S67. When all ORB key points have been traversed, obtain the depth-restored semantic image and pass it into the semantic segmentation thread for the semantic thread to repair the segmentation map.

[0073] S7. Use the remaining static key points for camera pose estimation or repositioning.

[0074] S8. In dense mapping, use semantic information and tracking information to establish a semantic point cloud map, further including:

[0075] S81. Obtain the key frame pose information, RGB-D image information, and its semantic segmentation map passed in by the tracking thread;

[0076] S82. Obtain each pixel P on the RGB image uv , if the semantic label of this pixel is a person, then do not add it to the point cloud map;

[0077] S83. Each pixel point P uv has the following transformation relationship with the map point, where T cw is the transformation matrix from the world coordinate system to the camera coordinate system, K is the camera internal parameter matrix, and Z is the depth value of the pixel point P uv .

[0078]

[0079] S84. Use the following formula to project the two-dimensional pixel point P uv into the three-dimensional space to obtain the coordinates of the map point P w .

[0080]

[0081] S85. Output the point cloud map.

[0082] In this embodiment, the comparison of the ATE (Absolute Pose Error) between the SLAM algorithm and the ORB-SLAM algorithm is shown in the following table:

[0083]

[0084]

[0085] In this embodiment, the ATE (absolute posture error) comparison between the SLAM algorithm and the DS-SLAM algorithm is shown in the following table:

[0086]

[0087] In this embodiment, under the improvement of the original ORB-SLAM2, not only the original real-time performance can be guaranteed, but also the dynamic points can be identified and eliminated in a high dynamic environment, thereby improving the accuracy of the SLAM system. At the same time, this method is also more accurate than DS-SLAM.

[0088] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A real-time semantic vSLAM algorithm based on depth map restoration, characterized by: The real-time semantic vSLAM algorithm comprises the following steps: S1. Input the RGB-D image frame taken by the depth camera, and pass the RGB-D image to the semantic segmentation thread and tracking thread respectively, and update the image to be segmented; S2, adopts a structure that does not block the tracking thread. The semantic thread uses DeeplabV3+ based on the MobileNetV2 framework to segment the latest image to be segmented and obtain the latest semantic segmentation map; S3, repairing the segmentation map, merging the segmentation map and the corresponding deep restored semantic image as the latest semantic segmentation frame; S4, convert the RGB image of the current frame into a grayscale image, and extract its ORB key points; S5, the tracking thread uses the latest semantic segmentation map without waiting for the semantic segmentation thread to end, and uses the LK multi-layer optical flow method to remove the dynamic key points of the current frame; S51, obtaining the ORB key points and grayscale images of the current frame and the latest segmented frame; S52, assuming that the grayscale of the point (x1, y1) in the latest segmented frame is I1(x1, y1), and the grayscale of the point (x2, y2) in the current frame is I2(x2, y2). Based on the assumption that the grayscale is constant, the grayscale of the same point is constant. The Gauss-Newton formula is used to track the movement of the ORB key points. The Gauss-Newton formula is: S6, restore the semantic image using ORB pairing points and depth information; S61, determining whether the ORB key point of the current frame is a dynamic point and whether it has not been traversed, if so, performing a depth traversal using a breadth-first algorithm; S62, create a new queue a, put the ORB key point at the end of the queue, and obtain the depth value of the point as the basic depth d of the traversal; S7, using the remaining static key points to estimate or relocate the camera's pose; S8. In dense mapping, semantic information and tracking information are used to build a semantic point cloud map.

2. The real-time semantic vSLAM algorithm based on depth map repair according to claim 1, characterized in that: The S2 includes: S21. Use libtorch to call the pt file of DeeplabV3+ and load the DeeplabV3+ semantic segmentation network model based on the MobileNetV2 framework. S22, when there is a latest image input, update the latest frame to be segmented; S23, after the semantic segmentation thread completes the segmentation of the previous image, it segments the latest image to be segmented.

3. The real-time semantic vSLAM algorithm based on depth map repair according to claim 2, characterized in that: The S5 further comprises: S53, using the original image as the bottom layer of the image pyramid, scaling the lower layer image by a certain factor each time going up a layer, to obtain images of different resolutions, and when calculating the optical flow tracking ORB key points, starting with the top layer image, and then using the grayscale calculation result of the previous layer as the initial value of the next layer to calculate, until the calculation reaches the bottom layer, completing the pairing of the ORB key points; S54, determining whether the semantic label of the paired point of the current image ORB key point on the latest segmented frame RGB image on its semantic mask is a dynamic object, and if so, marking the ORB key point as a dynamic key point and then removing it.

4. The real-time semantic vSLAM algorithm based on depth map repair according to claim 3, characterized in that: The S6 further comprises: S63, initialize the queue head=0, tail=1; S64, when head! = tail, head++, start traversing in the up, down, left, and right directions of the point corresponding to a[head], and record its depth value as d n (n=1,2,3,4); S65, if d-0.1<d n <d+0.1, then mark the point as a dynamic point, record it as 1, tail++, and put it at the end of the queue to be traversed; otherwise, mark it as a static point, record it as 0, and continue searching in the next direction.

5. The real-time semantic vSLAM algorithm based on depth map repair according to claim 4, characterized in that: The S6 further includes: S66, repeat the above steps S62 to S65 until all points in the traversal queue are traversed, and then proceed to the next ORB key point traversal; S67. After all ORB key points are traversed, a deeply restored semantic image is obtained and passed to the semantic segmentation thread for use by the semantic thread to repair the segmentation map.

6. The real-time semantic vSLAM algorithm based on depth map repair according to claim 1, characterized in that: The S8 includes: S81, obtaining key frame pose information, RGB-D image information and its semantic segmentation map transmitted by the tracking thread; S82, obtain each pixel P on the RGB image uv , if the semantic label of the pixel is human, it will not be added to the point cloud map.

7. The real-time semantic vSLAM algorithm based on depth map repair according to claim 6, characterized in that: The S8 further comprises: S83, each pixel point P uv The transformation relationship with the map point is as follows, where T cw is the transformation matrix from the world coordinate system to the camera coordinate system, K is the camera intrinsic parameter matrix, and Z is the pixel point P uv The depth value of S84, using the following formula to convert the two-dimensional pixel point P uv Project into three-dimensional space to get map point P w The coordinates of S85. Output the point cloud map.

Images