A robust pose estimation method based on instance segmentation in dynamic scenes

Abstract

The present invention discloses a robust pose estimation method based on instance segmentation in dynamic scenes, which can eliminate dynamic scenes in the visual navigation process. The idea is to detect dynamic areas in image sequences in combination with optical flow tracking in instance segmentation areas and eliminate them. The algorithm first uses instance segmentation combined with a target tracking algorithm to lock potential dynamic object areas; then uses the optical flow method to perform dynamic point detection in the segmented area to obtain the dynamic probability of objects in the segmented area; finally, the dynamic probability is combined with semantic prior information to determine the dynamic area. By eliminating the dynamic part in the environmental information, the present invention can significantly improve the real-time positioning accuracy of the SLAM system and greatly increase the robustness of the robot positioning system to meet the application scenarios of more intelligent robots.

Description

Technical Field

[0001] The present invention relates to the technical field of image graphics and a simultaneous positioning and mapping technology, and in particular to a robust pose estimation method based on instance segmentation in a dynamic scene. Background Art

[0002] In recent years, intelligent robots have become an important research direction in academia and industry. To make robots more intelligent and autonomous, they must improve their ability to recognize unknown environments. Simultaneously positioning and mapping (SLAM) is a key technology for building three-dimensional scene maps and positioning the machine itself. It has been successfully applied to products such as unmanned vehicles, drones, express delivery robots, and intelligent cleaning robots.

[0003] At present, some advanced open source SLAM systems can guarantee real-time operation and stable positioning. However, these systems are mostly based on the following assumptions: the environment in which the system operates is static. However, there are a large number of dynamic objects in the real environment we live in, and the trees in the environment often shake. If these motion features are involved in the solution, it will seriously affect the accuracy of pose solution and mapping. Especially when dynamic objects occupy a large detection scene, the system will have wrong data association, resulting in the deterioration of the entire SLAM process. Therefore, considering the factors of dynamic objects and removing them can significantly improve the real-time positioning accuracy of the SLAM system, and is also conducive to the construction of three-dimensional maps. Improving the perception ability of the robot, judging which parts of the acquired environmental information are dynamic, and eliminating them during the positioning process can greatly increase the robustness of the robot positioning system; on the other hand, removing dynamic objects can improve the accuracy of mapping and meet more intelligent application scenarios. Summary of the invention

[0004] To solve the above problems, the present invention discloses a robust pose estimation method based on instance segmentation in dynamic scenes to remove dynamic objects in images so that the SLAM system can increase the accuracy of pose solution and mapping.

[0005] To achieve the above object, the technical solution of the present invention is as follows:

[0006] A robust pose estimation method based on instance segmentation in a dynamic scene comprises the following steps:

[0007] (1) Use the real-time instance segmentation network Yolcat to perform instance segmentation to identify potential dynamic objects, and combine it with Camshift to track the identified objects to obtain the object segmentation results;

[0008] (2) Combined with the LK optical flow method, the dynamic probability of all feature points in the segmented object is calculated to obtain the dynamic probability p of the object d ;

[0009] (3) Preset the dynamic probability p of a specific category of objects w As the semantic prior probability, the dynamic probability p of the object d As the likelihood probability, the posterior probability p is determined by the Bayesian formula post , and set the threshold σ to determine whether the segmented area is a dynamic area;

[0010] (4) Eliminate feature points in the dynamic area, perform feature matching, and calculate the camera pose.

[0011] The present invention is further improved in that:

[0012] In step (1), Yolcat is used to perform instance segmentation to identify potential dynamic objects, and Camshift is used to track the identified objects to obtain the segmentation results of the objects. The specific steps are as follows:

[0013] (1.1) Extract the ORB feature points of the current frame;

[0014] (1.2) Use Yolact to identify and segment objects and label each object;

[0015] (1.3) Use Camshift to track each segmented object;

[0016] (1.4) Calculate the object area C tracked by Camshift in the next frame area Segment object region Y with Yolact area The regional overlap θ, where:

[0017]

[0018] (1.5) If the threshold ε is exceeded, the Yolact segmentation fails and the result tracked by Camshift is used; otherwise, the result of Yolact is used;

[0019] (1.6) Update the Camshift tracking object using the object obtained in step (1.5).

[0020] The present invention is further improved in that:

[0021] In step (2), the dynamic probability of all feature points in the segmented object is calculated by combining the LK optical flow method to obtain the dynamic probability p of the object. d , the specific steps are:

[0022] (2.1) Use the LK optical flow method to track the matching point pairs in the current frame;

[0023] (2.2) Eliminate invalid matching point pairs;

[0024] (2.3) The basic matrix F is calculated based on the camera internal parameters;

[0025] (2.4) Calculate the baseline I of the current frame according to the basic matrix F;

[0026] (2.5) Calculate the distance between the feature point and the epipolar line in the current frame. Take the average of the distance between all feature points and the epipolar line in the instance segmented object to obtain the dynamic probability p of the segmented object. d .

[0027] The present invention is further improved in that:

[0028] In step (3), the dynamic probability p of a specific category of objects is preset w As the semantic prior probability, the dynamic probability p of the object d As the likelihood probability, the posterior probability p is determined by the Bayesian formula post , and set the threshold σ to determine whether the segmented area is a dynamic area. The specific steps are:

[0029] (3.1) Query Table 1 to obtain the semantic dynamic probability p of a specific object w , as the prior probability;

[0030] (3.2) Substitute the dynamic probability p obtained in step (2) d As likelihood probability;

[0031] (3.3) Using the Bayesian formula, we get the posterior probability p of the observation post ;

[0032] (3.4) The posterior probability p post Compare with the set threshold σ to determine whether the segmented area is a dynamic area.

[0033] The present invention is further improved in that:

[0034] In step (4), the feature points in the dynamic area are removed, feature matching is performed and the camera pose is calculated. The specific steps are as follows: if the segmented area is determined to be a dynamic area in step (3), the feature points in the dynamic area are removed, and feature matching and camera pose calculation are performed again.

[0035] The beneficial effects of the present invention are:

[0036] In view of the fact that the current visual odometer based on the feature point method is often disturbed by unknown dynamic objects, resulting in large positioning errors, this paper proposes a pose solution method for removing dynamic area feature points. First, the Camshfit algorithm is used to track potential moving objects in the scene to obtain a more accurate target, and then the regional dynamic probability obtained by the Bayesian estimation and LK optical flow method is used to remove dynamic feature points.

[0037] In order to solve the problem of excessive positioning error of SLAM algorithm in dynamic scenes, the present invention proposes a positioning algorithm that can run stably and in real time in dynamic scenes, improves the accuracy of visual odometer pose solution, and can be widely used in scenes such as autonomous driving and indoor navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart of a robust pose estimation method based on instance segmentation in dynamic scenes;

[0039] Figure 2 This is the flow chart of Yolact instance segmentation and Camshift fusion;

[0040] Figure 3 This is the LK optical flow static point detection algorithm process;

[0041] Figure 4 Dynamic probability scoring results for some objects. DETAILED DESCRIPTION

[0042] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. It should be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the accompanying drawings, and the words "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0043] like Figure 1 Shown is a schematic flow chart of the method of the present invention.

[0044] Step S1: Use Yolcat to perform instance segmentation to identify potential dynamic objects, and combine Camshift to track the identified objects to obtain the object segmentation results. Specifically include:

[0045] S1.1. Extract the ORB feature points of the current frame: detect the FAST corner point positions, calculate the BRIEF descriptor based on the corner point positions, and obtain the feature points of the current frame;

[0046] S1.2. Use Yolact to identify and segment objects and label each object: set reasonable hyperparameters such as learning rate and number of single training samples, and train the CoCo dataset through the Yolact instance segmentation network to obtain the training model;

[0047] S1.3. Use Camshift to track each segmented object: determine the Yolact segmented object area, calculate the back projection map of the area, iterate the Mean-Shift algorithm according to the back projection map to obtain the center position of the segmented area, and enter the next frame for tracking;

[0048] S1.4. Calculate the object area C tracked by Camshift in the next frame area Segment object region Y with Yolact area The regional overlap θ, where:

[0049]

[0050] S1.5, if the set threshold ε is exceeded, the Yolact segmentation fails and the result tracked by Camshift is used, otherwise the result of Yolact is used;

[0051] S1.6. Update the Camshift tracking object using the object obtained in step (1.5).

[0052] Step S2: Combined with the LK optical flow method, calculate the dynamic probability of all feature points in the segmented object and obtain the dynamic probability p of the object d . Specifically include:

[0053] S2.1. Track the matching point pairs of the current frame using the LK optical flow method: calculate the optical flow pyramid and obtain the matching point pairs of the current frame;

[0054] S2.2, remove invalid matching point pairs: if the matching pair is too close to the edge of the image or the pixel difference of the 3*3 image block at the center of the matching pair is too large, the matching pair is removed;

[0055] S2.3. Calculate the basic matrix F based on the camera intrinsic parameters: first use the traditional camera calibration method to obtain the camera intrinsic parameters, extract the feature points of any two images from all the sequence images, and perform feature point matching; according to the feature point matching results, use the RANSAC (random sampling consensus) algorithm to obtain the basic matrix F and the essential matrix E;

[0056] S2.4. Calculate the baseline I of the current frame according to the basic matrix F:

[0057]

[0058] S2.5. Calculate the distance between the feature point and the epipolar line in the current frame:

[0059]

[0060] The size of D reflects the dynamic probability of the matched feature points. The average distance between all feature points and the polar line in the instance segmentation object is calculated to obtain the dynamic probability p of the object. d :

[0061]

[0062] Where N represents the number of feature points in the region.

[0063] Step S3: Preset the dynamic probability p of a specific category of objects w As the semantic prior probability, the dynamic probability p of the object d As the likelihood probability, the posterior probability p is determined by the Bayesian formula post , and set the threshold σ to determine whether the segmented area is a dynamic area. The specific steps are:

[0064] S3.1. The dynamic probability of an object in a real scene is scored from 0 (static) to 10 (dynamic), and the scoring result is used as the dynamic weight p w Assign segmented regions. Table 1 shows the potential motion probability of the set object. Figure 4 Shown are the scoring results for some representative objects, with buildings, tables, cars, and pedestrians being assigned dynamic weights of 0, 1, 7, and 9 respectively.

[0065]

[0066] Table 1S3.2, using Bayesian to fuse object dynamic probability and prior dynamic weight:

[0067]

[0068] Where S is the state of a region and Z is the observation information.

[0069] The p w As the prior probability, p d As likelihood, the above formula can be transformed into:

[0070]

[0071] Where λ is the scale factor, p post is the calculated posterior probability.

[0072] S3.3, set the threshold σ, and determine whether the segmented area is a dynamic area according to the following rules:

[0073]

[0074] Step S4: Eliminate feature points in the dynamic area, perform feature matching and calculate the camera pose.

[0075] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above technical features.

Claims

1. A robust pose estimation method based on instance segmentation in dynamic scenes, characterized in that: The steps include: (1) Use the real-time instance segmentation network Yolcat, You Only Look At Coefficients to perform instance segmentation to identify potential dynamic objects, and combine it with the continuously adaptive Mean-Shift algorithm Camshift, Continuously Adaptive Mean Shift algorithm to track the identified objects and obtain the object segmentation results; (2) Combined with LK and Lucas–Kanade optical flow methods, the dynamic probability of all feature points in the segmented object is calculated to obtain the dynamic probability p of the object. d ; In step (2), the dynamic probability of all feature points in the segmented object is calculated by combining the LK optical flow method to obtain the dynamic probability p of the object. d , the specific steps are: (2.1) Use the LK optical flow method to track the matching point pairs in the current frame; (2.2) Eliminate invalid matching point pairs; (2.3) The basic matrix F is calculated based on the camera internal parameters; (2.4) Calculate the baseline I of the current frame according to the basic matrix F; (2.5) Calculate the distance between the feature point and the polar line in the current frame and average the distance between all feature points and the polar line in the instance segmented object to obtain the dynamic probability p of the segmented object. d ; (3) Preset the dynamic probability p of a specific category of objects w As the semantic prior probability, the dynamic probability p of the object d As the likelihood probability, the posterior probability p is determined by the Bayesian formula post , and set the threshold σ to determine whether the segmented area is a dynamic area; (4) Eliminate feature points in the dynamic area, perform feature matching, and calculate the camera pose.

2. The method for robust pose estimation based on instance segmentation in a dynamic scene according to claim 1, characterized in that: In step (1), Yolcat is used to perform instance segmentation to identify potential dynamic objects, and Camshift is used to track the identified objects to obtain the object segmentation results. The specific steps are as follows: (1.1) Extract the ORB feature points of the current frame; (1.2) Use Yolact to identify and segment objects and label each object; (1.3) Use Camshift to track each segmented object; (1.4) Calculate the object area C tracked by Camshift in the next frame area Segment object region Y with Yolact area The regional overlap θ, where: (1.5) If the overlap θ exceeds the set threshold ε, it proves that the Yolact segmentation fails, and the result tracked by Camshift is used, otherwise the result of Yolact is used; (1.6) Update the Camshift tracking object using the object obtained in step (1.5).

3. The method for robust pose estimation based on instance segmentation in a dynamic scene according to claim 1, characterized in that: In step (3), the dynamic probability p of a specific category of objects is preset w As the semantic prior probability, the dynamic probability p of the object d As the likelihood probability, the posterior probability p is determined by the Bayesian formula post , and set the threshold σ to determine whether the segmented area is a dynamic area. The specific steps are: (3.1) According to the object potential dynamic probability table, obtain the dynamic probability p of a specific object w , as the semantic prior probability; (3.2) Substitute the dynamic probability p obtained in step (2) d As likelihood probability; (3.3) Using the Bayesian formula, we get the posterior probability p of the observation post ; (3.4) The posterior probability p post Compare with the set threshold σ to determine whether the segmented area is a dynamic area; 4. The method for robust pose estimation based on instance segmentation in a dynamic scene according to claim 1, characterized in that: In step (4), the feature points in the dynamic area are removed, feature matching is performed and the camera pose is calculated. The specific steps are as follows: if the segmented area is determined to be a dynamic area in step (3), the feature points in the dynamic area are removed, and feature matching and camera pose calculation are performed again.

Images