A visual SLAM method in dynamic environments based on instance segmentation

Abstract

The present invention provides a visual SLAM method in a dynamic environment based on instance segmentation, which mainly solves the problem that the accuracy of pose estimation of the visual SLAM method in a dynamic environment is seriously disturbed. The method introduces the SOLOV2 instance segmentation module, which processes the input image as an independent module and outputs the instance segmentation result. A data transmission method between the two is designed based on the HTTP communication protocol, so that the instance segmentation method is deployed relatively simply. The instance segmentation result is stored in a masked form, and in order to save memory usage, it is compressed in the RLE format. Using the image, depth image and instance segmentation result as input, a person-object connection detection method is designed for identifying semi-static targets. Based on semantic prior information and semi-static target detection results, dynamic targets and static targets are separated, and after removing the feature points obtained on the dynamic object, the accuracy of the estimated trajectory is significantly improved.

Description

Technical Field

[0001] The present invention belongs to the intersection field of computer vision, robotics, automation and artificial intelligence, and specifically relates to a visual SLAM method in a dynamic environment based on instance segmentation. Background Art

[0002] Simultaneous Localization And Mapping (SLAM), simultaneous positioning and map construction are hot issues in the field of mobile robots. Its two main goals are positioning and mapping. Positioning is to determine the state of the robot itself, including position, posture and other information; mapping is to build an environmental information description around the robot. The SLAM system can use a variety of sensors, such as lidar, camera, IMU, GPS, etc. This article discusses visual SLAM, which achieves positioning and mapping goals by processing the image information obtained by the camera. Visual SLAM plays an important role in augmented reality, virtual reality, household sweeping machines, drone navigation, autonomous driving and other fields.

[0003] At present, mainstream SLAM technology has achieved mature and stable use effects in static environments. The so-called static environment refers to the objects in the camera's field of view not moving. However, such a static environment assumption is invalid in the real environment. Moving objects such as pedestrians and vehicles are objects that frequently appear in daily environments. In addition, changes in illumination, such as changes in automatic exposure parameters caused by the camera entering a bright environment from a dark environment, and changes in light sources caused by day and night changes, will cause serious interference to the camera's pose estimation. In some articles, dynamic environment refers to an environment with dynamic objects, while in other articles it refers to an environment with changing illumination. This article mainly refers to an environment with dynamic objects. At present, the main idea for processing dynamic environments is to introduce deep learning technology into SLAM to extract information from images, such as semantic segmentation and instance segmentation. By removing dynamic objects in the environment, their impact on pose estimation can be reduced. However, there are currently widespread problems such as poor segmentation accuracy and speed, difficulty in deploying deep learning networks, and ignoring the impact of semi-static objects. Summary of the invention

[0004] The purpose of the present invention is to provide a visual SLAM method in a dynamic environment based on instance segmentation, which uses instance segmentation results to process the interference of moving objects and significantly improves the accuracy of pose estimation.

[0005] A visual SLAM method in a dynamic environment based on instance segmentation, comprising the following steps:

[0006] Step 1: Read the image sequence, store it in matrix format, and send the data to the instance segmentation module based on the HTTP communication protocol;

[0007] Step 2: In the instance segmentation module, the image matrix is ​​received and sent to the SOLOV2 network for prediction. The result is compressed and returned to the main program in JSON format.

[0008] Step 3: Send the image information, depth information, and instance segmentation results to the human-object connection detection method to detect semi-static objects;

[0009] Step 4: Use prior information and semi-static object detection results to divide the dynamic and static objects in the image;

[0010] Step 5: Extract ORB feature points from the image;

[0011] Step 6: Delete the ORB feature points extracted from the dynamic objects according to the dynamic and static object segmentation results;

[0012] Step 7: Match feature points between frames and track feature points;

[0013] Step 8: Use the EPNP method to solve the camera pose, record the camera pose at each moment, and obtain the camera trajectory.

[0014] Furthermore, the step 1 is specifically as follows:

[0015] The open source Opencv library is used to store image data in matrix form. The instance segmentation network SOLOV2 is used as an independent module. In order to ensure that the independent module runs asynchronously with the SLAM program, the image data is sent to the instance segmentation module based on HTTP (Hyper Text Transfer Protocol).

[0016] Furthermore, the step 2 is specifically as follows:

[0017] In the instance segmentation module, the server is started to wait for the image transmitted by the SLAM program. The result is compressed according to RLE (run-length encoding), that is, 00000111100111110... can be compressed into 54251..., which greatly compresses the memory usage without loss, and then returns the segmentation result to the main program in JSON ((JavaScript Object Notation)) method.

[0018] Furthermore, the step 3 is specifically as follows:

[0019] First, with the edge pixel as the center, set the radius to 3 pixels, and get a circle with 16 pixels. Secondly, check each pixel to see if there is another mask at its location and whether its depth value is close to the depth value of the center pixel. If more than 10 pixels meet the requirements, record that this edge pixel is connected to a person or an object. If more than 10 such edge pixels are found, record that there is a mask connected to a person. Set the category of the mask to a dynamic object, and then delete the feature points in it.

[0020] Furthermore, the step 4 is specifically as follows:

[0021] People are considered dynamic objects; animal labels: cats, dogs, birds, etc., and transportation categories: cars, trains, airplanes, etc. are also considered dynamic objects; the last detected category of semi-static objects is also considered a dynamic object in the detected frame.

[0022] Furthermore, the step 5 is specifically as follows:

[0023] ORB feature points are extracted by partitioning in the image. The quality of feature points can be reduced in areas where the number of extracted features is insufficient, and the extracted ORB feature points are cropped using a quadtree.

[0024] Furthermore, the step 6 is specifically as follows:

[0025] According to the dynamic and static object segmentation results, the objects in the image are divided into two categories: dynamic objects and static objects. The feature points in the dynamic object mask are removed, and only the feature points in the static environment are retained.

[0026] Furthermore, the step 7 is specifically as follows:

[0027] Match feature points between frames and use the random sampling consensus method to remove incorrect matching point pairs. The specific steps of the RANSAC method are as follows:

[0028] Step 7.1: Randomly select 4 sets of non-collinear feature point pairs in the data set and calculate the transformation matrix H;

[0029] Step 7.2: Use the obtained model as a test template and test other data points one by one. If the error is within the threshold, the point is considered to be an inlier. The model with the most inliers is the best model and is kept;

[0030] Step 7.3: Repeat steps 1 and 2, and after a certain number of iterations, use the internal points of the model to solve the final model parameters;

[0031] Step 7.4: Finally, estimate the inliers and the model error rate to evaluate the final model.

[0032] Calculation of the H matrix:

[0033]

[0034] Where (x, y) represents the corner position of the target image, (x', y') represents the corner position of the scene image, and s is the scale parameter.

[0035]

[0036] Furthermore, the step 8 is specifically as follows:

[0037] After removing the feature points obtained on the dynamic objects, only the feature points of the static objects and the environment are used for pose estimation. After matching the feature points between the two frames, these feature points are used to estimate the pose of the camera. The method used is EPnP, which makes fuller use of the information of the points and reduces the negative impact of data noise. The camera pose at each moment is recorded to obtain the camera trajectory.

[0038] Each point can be represented by 4 control points in the following form:

[0039]

[0040] Camera coordinate system:

[0041]

[0042] Convert known control point coordinates between coordinate systems:

[0043]

[0044] remember Simplify the three equations into two equations:

[0045]

[0046]

[0047] is the 3D reference point, The superscripts represent different coordinate systems, where w is the abbreviation of world and c is the abbreviation of camera. The four selected control points are The meaning of superscript letters is the same as that of 3D reference points.

[0048] Using the constraint that the spatial relationship between points does not change after the external parameter transformation: To confirm the most

[0049] The coordinates of the final control point in the camera coordinate system.

[0050] The beneficial effects of the present invention are:

[0051] (1) The method of the present invention reduces the difficulty of deploying instance segmentation methods and has good real-time performance;

[0052] (2) The present invention identifies semi-static objects and makes a good classification of dynamic, static, and semi-static objects;

[0053] (3) The present invention can operate in a high dynamic environment and has a high accuracy in pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A block diagram of a visual SLAM method in a dynamic environment based on instance segmentation proposed by the present invention;

[0055] Figure 2 This is a framework diagram of the instance segmentation module SOLOV2 method in the present invention;

[0056] Figure 3 It is a data transmission diagram of SOLOV2 and SLAM method in the present invention;

[0057] Figure 4 A method for detecting connection between a person and an object in the present invention;

[0058] Figure 5 The image sequence and instance segmentation result diagram used in the test of the present invention;

[0059] Figure 6 It is the data set trajectory evaluation diagram obtained by ORB-SLAM2 in the present invention;

[0060] Figure 7 This is a dataset trajectory evaluation diagram obtained by the method in the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described below in conjunction with the accompanying drawings.

[0062] The present invention is a visual SLAM method in a dynamic environment based on instance segmentation. The method block diagram is as follows: Figure 1 As shown, it includes the following steps:

[0063] Step 1: Read the image sequence, store it in matrix format, and send the data to the instance segmentation module based on the HTTP communication protocol;

[0064] The image sequence is used as the input of the method, and the open source Opencv library is used to store the image data in matrix form. The method uses the instance segmentation network SOLOV2 as an independent module. In order to ensure that the independent module runs asynchronously with the SLAM program, the image data is sent to the instance segmentation module based on HTTP (Hyper Text Transfer Protocol).

[0065] Step 2: In the instance segmentation module, the image matrix is ​​received and sent to the SOLOV2 network for prediction. The result is compressed and returned to the main program in JSON format.

[0066] like Figure 2 As shown in the figure, SOLOV2 transforms the instance segmentation problem into a classification problem in the method framework, and uses the constraints on position and shape to distinguish the instance categories of objects. If two objects with exactly the same shape and position appear in an image, then they are the same instance. Otherwise, they are two different instances. It greatly reduces the memory overhead of training and testing while maintaining consistent performance. Figure 3 As shown, in the instance segmentation module, the method starts the server to wait for the image transmitted by the SLAM program, and compresses the result according to RLE (run-length encoding), that is, run-length encoding. 00000111100111110... can be compressed into 54251..., which greatly compresses the memory usage without loss, and then returns the segmentation result to the main program in JSON ((JavaScript Object Notation)) method.

[0067] Step 3: Send the image information, depth information, and instance segmentation results to the human-object connection detection method to detect semi-static objects.

[0068] In some cases, static objects may be moved by people. For example, a book on a desk becomes a dynamic object when someone holds it in their hand. This leads to incorrect segmentation of feature points. These points from dynamic targets will reduce the performance of camera pose estimation. Therefore, it is necessary to process this type of object.

[0069] This method designs a method to detect the connection between people and objects. Figure 4 As shown, similar to FAST feature points, each edge pixel of the person's mask will be checked by the following steps.

[0070] First, we take the edge pixel as the center and set the radius to 3 pixels, which gives us a circle with 16 pixels. Secondly, we check each pixel to see if there is another mask at its location and if its depth value is close to the depth value of the center pixel. If more than 10 pixels meet our requirements, we record that this edge pixel is connected to a person or an object. If we find more than 10 such edge pixels, we record that there is a mask connected to a person. We set the category of the mask to a dynamic object and then delete the feature points in it.

[0071] Step 4: Use prior information and semi-static object detection results to divide the dynamic and static objects in the image;

[0072] The instance segmentation network used in this method uses the COCO dataset in the training process, which contains 80 types of objects. The objects contained therein are rich in variety and are common objects in daily life. Among them, the possibility that people are dynamic objects is very high, so people are regarded as dynamic objects; animal labels: cats, dogs, birds, etc., and transportation categories: cars, trains, airplanes, etc. are also considered dynamic objects; finally, there are categories of detected semi-static objects, which are also considered dynamic objects in the detected frames.

[0073] Step 5: Extract ORB feature points from the image;

[0074] ORB feature points are extracted by partitioning in the image. The quality of feature points can be reduced in areas where the number of extracted features is insufficient, and the extracted ORB feature points are cropped using a quadtree to ensure the quantity and quality of feature points.

[0075] Step 6: Delete the ORB feature points extracted from the dynamic objects according to the dynamic and static object segmentation results;

[0076] According to the dynamic and static object segmentation results, the objects in the image can be divided into two categories: dynamic objects and static objects. By removing the feature points in the dynamic object mask, only the feature points in the static environment can be retained.

[0077] Step 7: Match feature points between frames and track feature points;

[0078] Match feature points between frames and use the random sample consensus (RANSAC) method to remove incorrect matching point pairs. The specific steps of the RANSAC method are as follows:

[0079] 1. Randomly select 4 sets of non-collinear feature point pairs in the data set and calculate the transformation matrix H.

[0080] 2. Use the obtained model as a test template and test other data points one by one. If the error is within the threshold, the point is considered to be an inlier. The model with the most inliers is the best model and is kept.

[0081] 3. Repeat steps 1 and 2, and after a certain number of iterations, use the internal points of the model to solve the final model parameters.

[0082] 4. Finally, estimate the error rate of the in-game points and the model to evaluate the final model.

[0083] The calculation method of H matrix is ​​as follows

[0084]

[0085] Where (x, y) represents the corner position of the target image, (x', y') represents the corner position of the scene image, and s is the scale parameter.

[0086]

[0087] The error calculation is shown in the above formula.

[0088] Step 8: Use the EPNP method to solve the camera pose, record the camera pose at each moment, and obtain the camera trajectory.

[0089] This method removes the feature points obtained on dynamic objects and only uses the feature points of static objects and the environment for pose estimation. After matching the feature points between two frames, these feature points are used to estimate the pose of the camera. The method used is EPnP (Efficient Perspective-n-Point), which makes fuller use of the information of the points and reduces the negative impact of data noise. The camera pose is recorded at each moment to obtain the camera trajectory.

[0090] The EPnP method uses the world coordinates of n known 3D points, selects four control points with known world coordinates through principal component analysis (PCA), and establishes a new local coordinate system, thereby expressing the world coordinates of the 3D points with new control points (world coordinates). Then, using the camera projection model and the 2D point pixel coordinates, a linear equation system about the coordinates of the control points in the camera coordinate system is established, and the coordinates of the four control points in the camera coordinate system are solved, and then the coordinates of the n 3D points in the camera coordinate system are solved. In this way, the spatial points correspond to each other in the two coordinate systems, and the problem is converted into a 3D-3D problem, which can be solved using ICP.

[0091] The 3D reference point is The superscripts represent different coordinate systems, where w is the abbreviation of world and c is the abbreviation of camera. The four selected control points are The meaning of superscript letters is the same as that of 3D reference points.

[0092] Each point can be represented by 4 control points in the following form:

[0093]

[0094] Similarly, in the camera coordinate system, there are also:

[0095]

[0096] The known control point coordinates need to be converted between coordinate systems.

[0097]

[0098] remember Simplify the three equations into two equations:

[0099]

[0100]

[0101] Using the constraint that the spatial relationship between points does not change after the external parameter transformation: Used to confirm the coordinates of the final control point in the camera coordinate system.

[0102] The present invention introduces the SOLOV2 instance segmentation module, which processes the input image as an independent module and outputs the instance segmentation result. A data transmission method between the two is designed based on the HTTP communication protocol, so that the instance segmentation method is deployed relatively simply. The instance segmentation result is stored in a masked form, and in order to save memory usage, it is compressed in the RLE format. Taking the image, depth image and instance segmentation result as input, a person-object connection detection method is designed to identify semi-static targets. Based on semantic prior information and semi-static target detection results, dynamic targets and static targets are separated. After removing the feature points obtained on the dynamic object, the accuracy of the estimated trajectory is significantly improved.

[0103] The present invention uses a data set to conduct experiments. A dynamic video sequence in the TUM data set is used. The content and instance segmentation results are shown in Figure 5 As shown, the present invention is compared with the existing ORB-SLAM2 method.

[0104] from Figure 6It can be seen that there is a significant error between the trajectory (colored line) shown in the figure and the true trajectory (black dotted line), indicating that ORB-SLAM2 cannot handle moving objects in a dynamic environment very well. These objects have a significant interference with the pose estimation, and the estimated trajectory has a significant deviation. The absolute trajectory error (ATE) is 0.7521

[0105] from Figure 7 It can be seen that the estimated trajectory (colored line) and the reference true value trajectory (black dashed line) overlap well without obvious deviation. The error size can be intuitively compared according to the color of the line. The absolute trajectory error is 0.0180, which is significantly better than ORB-SLAM2.

[0106] In summary, in order to solve the problem of low pose estimation accuracy of existing SLAM methods in dynamic environments, the present invention proposes a visual SLAM method in dynamic environments based on instance segmentation. Experimental results show that compared with the existing ORB-SLAM2 method, the present invention can operate better in dynamic environments and has higher pose estimation accuracy.

[0107] As a basic task of computer vision, instance segmentation can accurately predict the pixels contained in an object and distinguish multiple independent individuals in the same type of objects, which is convenient for tracking the trajectory of each object. Instance segmentation technology can provide object-level semantic information for the SLAM system, so that each module in the entire SLAM system can be fully optimized for dynamic environments. This paper uses SOLOV2 for instance segmentation to obtain good segmentation accuracy, and quickly deploys it as an independent module to interact with the SLAM system through network communication protocols. Furthermore, the results obtained by instance segmentation technology are used to detect semi-static objects and separate dynamic objects, which enhances the stability of the existing ORB-SLAM2 system in an environment containing dynamic objects and improves the accuracy of pose estimation. After processing the dynamic environment in the above way, the obstacles to the application of visual SLAM systems in real environments will be greatly reduced, and it has a high application value.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A visual SLAM method in a dynamic environment based on instance segmentation, characterized in that: The following steps are involved: Step 1: Read the image sequence, store it in matrix format, and send the data to the instance segmentation module based on the HTTP communication protocol; Step 2: In the instance segmentation module, the image matrix is ​​received and sent to the SOLOV2 network for prediction. The result is compressed and returned to the main program in JSON format. Step 3: Send the image information, depth information, and instance segmentation results to the human-object connection detection method to detect semi-static objects; Step 4: Use prior information and semi-static object detection results to divide the dynamic and static objects in the image; Step 5: Extract ORB feature points from the image; Step 6: Delete the ORB feature points extracted from the dynamic objects according to the dynamic and static object segmentation results; Step 7: Match feature points between frames and track feature points; Step 8: Use the EPNP method to solve the camera pose, record the camera pose at each moment, and obtain the camera trajectory.

2. A visual SLAM method in a dynamic environment based on instance segmentation according to claim 1, characterized in that: The step 1 is specifically as follows: The open source Opencv library is used to store image data in matrix form. The instance segmentation network SOLOV2 is used as an independent module. In order to ensure that the independent module runs asynchronously with the SLAM program, the image data is sent to the instance segmentation module based on HTTP (Hyper Text Transfer Protocol).

3. A visual SLAM method in a dynamic environment based on instance segmentation according to claim 1, characterized in that: The step 2 is specifically as follows: In the instance segmentation module, the server is started to wait for the image transmitted by the SLAM program. The result is compressed according to RLE (run-length encoding), that is, 00000111100111110... can be compressed into 54251..., which greatly compresses the memory usage without loss, and then returns the segmentation result to the main program in JSON ((JavaScript Object Notation)) method.

4. A visual SLAM method in a dynamic environment based on instance segmentation according to claim 1, characterized in that: The step 3 is specifically as follows: First, with the edge pixel as the center, set the radius to 3 pixels, and get a circle containing 16 pixels; secondly, check each pixel to see if there is another mask at its position and whether its depth value is close to the depth value of the center pixel; if more than 10 pixels meet the requirements, record that this edge pixel is connected to people and objects; if more than 10 such edge pixels are found, record that there is a mask connected to a person; set the category of the mask to dynamic object, and then delete the feature points in it.

5. A visual SLAM method in a dynamic environment based on instance segmentation according to claim 1, characterized in that: The step 4 is specifically as follows: People are considered dynamic objects; animal labels: cats, dogs, birds, and transportation categories: cars, trains, and airplanes are also considered dynamic objects; the last detected category of semi-static objects is also considered a dynamic object in the detected frame.

6. A visual SLAM method in a dynamic environment based on instance segmentation according to claim 1, characterized in that: The step 5 is specifically as follows: ORB feature points are extracted by partitioning in the image. The quality of feature points can be reduced in areas where the number of extracted features is insufficient, and the extracted ORB feature points are cropped using a quadtree.

7. A visual SLAM method in a dynamic environment based on instance segmentation according to claim 1, characterized in that: The step 6 is specifically as follows: According to the dynamic and static object segmentation results, the objects in the image are divided into two categories: dynamic objects and static objects. The feature points in the dynamic object mask are removed, and only the feature points in the static environment are retained.

8. A visual SLAM method in a dynamic environment based on instance segmentation according to claim 1, characterized in that: The step 7 is specifically as follows: Match feature points between frames and use the random sampling consensus method to remove incorrect matching point pairs; the specific steps of the RANSAC method are as follows: Step 7.1: Randomly select 4 sets of non-collinear feature point pairs in the data set and calculate the transformation matrix H; Step 7.2: Use the obtained model as a test template and test other data points one by one. If the error is within the threshold, the point is considered to be an inlier. The model with the most inliers is the best model and is kept. Step 7.3: Repeat steps 1 and 2, and after a certain number of iterations, use the internal points of the model to solve the final model parameters; Step 7.4: Finally, estimate the error rate of the inliers and the model to evaluate the final model; Calculation of the H matrix: Where (x, y) represents the corner position of the target image, (x', y') represents the corner position of the scene image, and s is the scale parameter; The error calculation is shown as follows:

9. A visual SLAM method in a dynamic environment based on instance segmentation according to claim 1, characterized in that: The step 8 is specifically as follows: After removing the feature points obtained on the dynamic objects, only the feature points of the static objects and the environment are used for pose estimation; after matching the feature points between the two frames, these feature point pairs are used to estimate the pose of the camera; the method used is EPnP, which makes full use of the point information and reduces the negative impact of data noise; the camera pose at each moment is recorded to obtain the camera trajectory; Each point can be represented by 4 control points in the following form: Camera coordinate system: Convert known control point coordinates between coordinate systems: remember Simplify the three equations into two equations: i=1,2,...,n are 3D reference points, i=1,2,...,n, the superscripts represent their coordinate systems, where w is the abbreviation of world and c is the abbreviation of camera; the four selected control points are j=1,2,3,4, j=1,2,3,4, the meaning of the superscript letters is the same as that of the 3D reference points; Using the constraint that the spatial relationship between points does not change after the external parameter transformation: Used to confirm the coordinates of the final control point in the camera coordinate system.

Images