A Method for Removing Dynamic Feature Points of Monocular Cameras Based on Lightweight Instance Segmentation

Abstract

A method for removing dynamic feature points of a monocular camera based on lightweight instance segmentation. 1. Correct the image according to the camera model parameters and extract feature points, and match the extracted feature points with the feature points obtained in the previous frame. 2. Open a new thread and synchronize with step 1 to input the image into the trained segmentation network for instance segmentation to obtain an instance mask. 3. Use the RANSAC algorithm to obtain the pose transformation of each mask and the area outside the mask, judge the dynamic mask and record it. 4. Calculate the dynamic probability of the feature point under this constraint according to the distance from the feature point to the nearest point of the dynamic mask. 5. Calculate the dynamic probability of the feature point under this constraint according to the epipolar constraint. 6. Use the Naive Bayes method to fuse the dynamic probabilities calculated in steps 4 and 5 above to calculate the dynamic probability corresponding to the feature point. The present invention can be applied to exclude the interference of dynamic objects in monocular vision SLAM navigation and improve the mapping and positioning accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of dynamic feature point elimination of a monocular camera, and specifically relates to a method for eliminating dynamic feature points of a monocular camera based on lightweight instance segmentation. Background Art

[0002] With the rapid development of electronic information technology and artificial intelligence technology, intelligent products have increasingly appeared in people's daily lives and play a crucial role in both public services and household daily life. Mobile robots need to use key technologies such as positioning, navigation, and decision-making when working autonomously in an unfamiliar environment. Among them, positioning is the most basic technology, and at the same time, the simultaneous localization and mapping technology (SLAM) is one of the most active research topics in robot positioning technology. During the process of feature point matching and pose estimation based on two consecutive image frames at the front end of visual SLAM, dynamic objects will cause pose estimation errors, resulting in large cumulative errors during long-term positioning. To solve this problem, most current solutions perform instance segmentation through image processing and determine and eliminate dynamic feature points based on prior experience. However, using only the method of instance segmentation to eliminate dynamic feature points is too dependent on the accuracy of instance segmentation. The speed of high-precision instance segmentation is difficult to meet real-time requirements, and low-precision instance segmentation will lead to incomplete elimination of dynamic feature points due to inaccurate instance masks. Feature points extracted at the edges of dynamic objects and feature points on dynamic objects without corresponding models in the instance segmentation network cannot be directly eliminated through the method of instance segmentation. Therefore, a method based on lightweight instance segmentation that integrates various criteria can be used to eliminate dynamic feature points.

[0003] The current existing technologies are as follows:

[0004] Application No. CN202110726415.7, patent name: A VO Weighted Optimization Method Based on a Monocular Camera, discloses a VO weighted optimization method based on a monocular camera. 1) Extract and match feature points from an image through KLT, eliminate mismatched points through RANSAC, and perform key frame screening; 2) Obtain the initial values of the positions of feature points, the inter-frame pose, and the inverse depth through the eight-point method and triangulation; 3) Calculate the photometric error and the inverse depth error to obtain the initial value of the weight; 4) Construct a least squares problem to prepare for optimizing variables, the positions of feature points and the pose of the camera; 5) Perform iterative operations to obtain the optimal solution and obtain the positions of feature points and the pose of the camera. The method of the present invention weights the feature points of each frame by utilizing the characteristics that the pixel difference and the spatial inverse depth difference of the matched feature points follow a t-distribution, and tries to reduce the influence of feature points with large pixel differences and at relatively near and far distances within the camera's field of view on the optimal solution of the global optimization.

[0005] This patent uses photometric error and inverse depth error to obtain the weighted values of each feature point in the key frame, and weights the feature point optimization terms during backend optimization to reflect the importance of each feature point; compared with this patent, this patent pre-segments the image, resolves the dynamic region through epipolar constraint and calculates the dynamic probability of the feature points around the dynamic region, sets a threshold to determine its dynamic probability, and removes the dynamic feature points at the front end. Since adding dynamic feature points to the backend will have an adverse impact on the system accuracy, removing them at the front end will have a better effect.

[0006] Application No.: CN201910444196.6, Patent Title: A Pose Estimation Method for Monocular Vision Cameras in Dynamic Environments

[0007] This patent first extracts and matches feature points and uses epipolar constraint to judge the dynamic characteristics of feature points, uses the Faster R-CNN method with a mask segmentation branch added to segment the image, uses the proportion of dynamic points in the segmented region to judge the dynamic characteristics of the whole region, and finally removes the feature points in the dynamic region; compared with this patent, this patent uses YOLACT to segment the image, opens a new thread to segment the image synchronously when extracting and tracking feature points to speed up the running speed. After judging the dynamic region, since the gray level changes greatly at the edge of the image segmentation region, the dynamic probability of the feature points outside the edge of the segmented region is calculated through epipolar constraint and mask constraint, and the dynamic points among them are removed, which is more accurate in removing dynamic feature points than the method of this patent. Summary of the Invention

[0008] Aiming at the existing problems, the present invention proposes a method for removing dynamic feature points of a monocular camera based on lightweight instance segmentation. This method can effectively remove the adverse impact of dynamic feature points on pose estimation and improve the accuracy and stability of the system. This method uses a semantic segmentation algorithm to perform instance segmentation on the visual image input by the camera to obtain a potential dynamic object mask, judges the dynamic characteristics of each mask through the RANSAC algorithm, and finally judges the dynamic probability of each feature point by fusing the dynamic probabilities of each feature point under mask constraint and epipolar constraint, and sets a threshold to remove the dynamic feature points to reduce the adverse impact of dynamic objects.

[0009] To achieve the above object, the technical solution adopted by the present invention is:

[0010] A method for removing dynamic feature points of a monocular camera based on lightweight instance segmentation, characterized by including the following steps:

[0011] Step 1: Use the calibrated camera parameters to correct the distortion of the input camera image, extract the ORB feature points of the image to obtain the ORB feature points of the image, and perform feature point matching with the ORB feature points of the previous frame image;

[0012] Step 2: While the feature point extraction in Step 1 is in progress, a new thread is opened to synchronously perform the following operations: Input the image sequence after image distortion correction into the trained YOLACT instance segmentation network, perform instance segmentation on the dynamic tendency objects such as people and vehicles in the input camera image, obtain the masks of each potential dynamic object and store them, denoted as M i , i ∈ 1, 2, …, n, where n is the total number of masks in the current frame;

[0013] Step 3: For the feature points within each potential dynamic object mask obtained in Step 2 and matching the feature points in the previous frame, use the RANSAC algorithm to calculate the homography matrix corresponding to each mask as H i , i ∈ 1, 2, …, n;

[0014] Use the remaining feature points except those within the mask to calculate their pose transformation as H using the RANSAC algorithm. Set a threshold γ. When the difference between H i , i ∈ 1, 2, …, n and H is greater than the threshold γ, it is determined that the object corresponding to this mask is in a moving state. Mark the feature points within the determined dynamic mask as dynamic feature points, which do not participate in pose estimation to eliminate their adverse effects on pose estimation;

[0015] Among the remaining feature points, there are still dynamic feature points that are not completely removed due to reasons such as low image segmentation accuracy and the models of corresponding objects not included in the network model. Record the motion states corresponding to each mask for subsequent judgment;

[0016] Step 4: Calculate the dynamic probability of the feature point under this constraint according to the distance from the feature point to the nearest point of the dynamic mask;

[0017] Step 5: Calculate the dynamic probability of the feature point under this constraint according to the epipolar constraint;

[0018] Step 6: Use the Naive Bayes method to fuse the dynamic probabilities calculated in Step 4 and Step 5 above to calculate the dynamic probability corresponding to this feature point;

[0019] Step 7: Finally, by setting a threshold, the feature points with the fused dynamic probability greater than this threshold are used as dynamic feature points, and the feature points less than this threshold are used as static feature points. Repeat the above steps in the subsequent images until all the image inputs of the camera are processed.

[0020] As a further improvement of the present invention, for the RANSAC algorithm in Step 3, the specific steps are as follows:

[0021] The first step is to randomly select 4 matching feature points to solve the following equation:

[0022] p2×H21 *p1 = 0

[0023] where p1 and p2 are the homogeneous coordinates of the matched feature points, which are 3×1 vectors, and H 21 is a 3×3 homography matrix, and the homography matrix is obtained after solving.

[0024] In the second step, a threshold is set, and p2×H 21 *p1 is calculated according to the homography matrix obtained in the first step. If the calculated value is less than the set threshold, the point is an inlier; otherwise, the point is an outlier. Count the number of inliers and outliers corresponding to this model among the matched feature points.

[0025] In the third step, compare the number of inliers of the current model with that of the previous best model. If the number of inliers of the current model is more than that of the previous best model, update the best model to the current model and record the corresponding number of inliers.

[0026] In the fourth step, repeat the above three steps until the iteration ends or the number of inliers corresponding to the current model is more than a certain number.

[0027] As a further improvement of the present invention, step 4 is specifically as follows:

[0028] Calculate the distance from the feature points outside the dynamic mask determined in step 3 in the image to the nearest dynamic mask edge, specifically:

[0029]

[0030] where u i , v i are respectively the abscissa and ordinate of the pixel coordinates of the i-th feature point, and P j (x), P j (y) are respectively the abscissa and ordinate of the pixel coordinates of the nearest dynamic mask to this feature point;

[0031] According to the distance of the feature points from the dynamic mask, calculate the dynamic possibility of each feature point according to the following formula:

[0032]

[0033] where λ m is a proportionality factor of the pixel distance. The smaller the distance of the feature point from the mask, the greater the corresponding dynamic probability.

[0034] As a further improvement of the present invention, step 5 is specifically as follows:

[0035] According to the epipolar constraint, first calculate the epipolar lines corresponding to the feature points in the previous adjacent frame in the current frame and calculate the distance between the corresponding feature points in the current frame and the epipolar lines, which is:

[0036]

[0037] wherein, respectively represent the homogeneous coordinates of the i-th feature point matched in the current frame and the previous frame, and F is the fundamental matrix, which can be expressed as:

[0038] F = K -T t ^ RK -1

[0039] where K is the camera internal parameter, R and t are respectively the rotation and translation of the camera relative to the previous frame, ^ represents the skew-symmetric matrix, and F can also be obtained by the following formula in actual calculation:

[0040]

[0041] where p1 and p2 are the matched feature points in the area where the feature point is located. To calculate the F matrix, eight matched points are required, namely the classic eight-point method;

[0042] The dynamic probability of the feature point determined by the epipolar constraint is obtained as:

[0043]

[0044] wherein, λ e is the scale factor. The greater the distance of the feature point from the epipolar line, the greater the dynamic probability.

[0045] As a further improvement of the present invention, step 6 is specifically as follows:

[0046] For the dynamic probabilities under each constraint obtained in steps 4 and 5, the proportion of the two in the fusion can be adjusted by adjusting λ m and λ e , and finally the two are fused using the Naive Bayes method. From the posterior probability formula of Naive Bayes:

[0047]

[0048] wherein, c represents the possible category of the sample, x represents the sample. From the above formula, for the hypothesis S i =1 indicates that the state of the i-th feature point is dynamic. The following formula is used to represent the dynamic probabilities obtained in steps 4 and 5 of the fusion:

[0049]

[0050] wherein, η is the prior fusion probability, representing the dynamic probability of this matched point in the previous frame. If there is no matched point, it is set to 0.5. In this way, the prior information obtained in the previous frame can be fused into the current frame through the set η.

[0051] The advantages of the present invention compared with the prior art are as follows:

[0052] Most of the current solutions for dynamic feature point elimination identify prior dynamic objects through technical means such as object recognition or instance segmentation and eliminate the corresponding feature points, relying on prior experience and the network accuracy of object recognition and instance segmentation. After the present invention uses relatively fast instance segmentation to first identify the dynamic region and then eliminate the feature points within the dynamic region, it fuses the dynamic edge and epipolar constraint to eliminate the dynamic feature points outside the dynamic mask, integrating a variety of technical means, making the dynamic feature point elimination accuracy higher and improving the positioning accuracy of the SLAM system. Compared with the method that relies on network segmentation accuracy, this method does not completely rely on prior experience and can use a faster segmentation network to improve the system speed. Brief Description of the Drawings

[0053] Figure 1 It is a system flow chart. Detailed Embodiment

[0054] The present invention will be further described in detail below in conjunction with the drawings and the specific embodiments:

[0055] As Figure 1 shown, the present invention provides a method for eliminating dynamic feature points of a monocular camera based on lightweight instance segmentation. Before performing this method, the SLAM system has completed the initialization step. The specific steps are as follows:

[0056] Step 1: Use the calibrated camera parameters to correct the distortion of the input camera image, extract the ORB feature points of the image to obtain the ORB feature points of the image, and perform feature point matching with the ORB feature points of the previous frame image.

[0057] Step 2: While the feature point extraction work in Step 1 is in progress, open a new thread to synchronously perform the following work: Input the image sequence after image distortion correction into the trained YOLACT instance segmentation network, perform instance segmentation on the objects with dynamic tendencies such as people and vehicles in the input camera image, obtain the masks of each potential dynamic object and store them, denoted as M i (i ∈ 1, 2, …, n), where n is the total number of masks in the current frame.

[0058] Step 3: For the feature points within each potential dynamic object mask obtained in Step 2 and that match the feature points in the previous frame, use the RANSAC algorithm to calculate the homography matrix corresponding to each mask as H i (i ∈ 1, 2, …, n). The specific steps are as follows:

[0059] The first step is to randomly select 4 matching feature points to solve the following equation:

[0060] p2×H 21 *p1 = 0

[0061] where p1 and p2 are the homogeneous coordinates of the matched feature points, which are 3×1 vectors, and H 21 is a 3×3 homography matrix. After solving, the homography matrix is obtained.

[0062] In the second step, set a threshold. According to the homography matrix obtained in the first step, calculate p2×H 21 *the value of p1. If the obtained value is less than the set threshold, then this point is an inlier; otherwise, this point is an outlier. Count the number of inliers and outliers corresponding to this model among the matched feature points.

[0063] In the third step, compare the number of inliers of the current model with that of the previous best model. If the number of inliers of the current model is more than that of the previous best model, then update the best model to the current model and record the corresponding number of inliers.

[0064] In the fourth step, repeat the above steps 1 to 3 until the iteration ends or the number of inliers corresponding to the current model is more than a certain number.

[0065] Use the feature points remaining except for the feature points inside the mask to calculate the pose transformation as H using the RANSAC algorithm. Set a threshold γ. When the difference between H i (i ∈ 1, 2, …, n) and H is greater than the threshold γ, it is determined that the object corresponding to this mask is in a moving state. Mark the feature points determined to be inside the dynamic mask as dynamic feature points, which do not participate in pose estimation and eliminate their adverse effects on pose estimation.

[0066] Among the remaining feature points, there are still dynamic feature points that are not completely removed due to reasons such as low image segmentation accuracy and the model of the corresponding object not included in the network model. Record the motion state corresponding to each mask for subsequent judgment.

[0067] Step 4: Calculate the distance from the feature points outside the dynamic mask determined in Step 3 in the image to the nearest edge of the dynamic mask. Specifically:

[0068]

[0069] where u i , v i are respectively the abscissa and ordinate of the pixel coordinates of the i-th feature point, and P j (x), P j (y) are respectively the abscissa and ordinate of the pixel coordinates of the nearest dynamic mask to this feature point.

[0070] According to the distance between feature points, the dynamic mask distance can calculate the dynamic probability of each feature point according to the following formula:

[0071]

[0072] Among them, λ m is a proportionality factor of the pixel distance. The smaller the distance between the feature point and the mask distance, the greater the corresponding dynamic probability.

[0073] Step 5: According to the epipolar constraint, first calculate the epipolar lines corresponding to the feature points in the previous adjacent frame in the current frame and calculate the distance between the corresponding feature points in the current frame and the epipolar lines, which is:

[0074]

[0075] Among them, respectively represent the homogeneous coordinates of the i-th feature point matched in the current frame and the previous frame. F is the fundamental matrix, which can be expressed as:

[0076] F = K -T t^RK -1

[0077] Among them, K is the camera internal parameter, R and t are the rotation and translation of the camera relative to the previous frame respectively, ^ represents the skew-symmetric matrix, and F can also be obtained through the following formula in actual calculation:

[0078]

[0079] Among them, p1 and p2 are the matched feature points in the region where the feature points are located. Eight matched points are required to calculate the F matrix, that is, the classic eight-point method.

[0080] In this way, the dynamic probability of the feature points determined by the epipolar constraint can be obtained as:

[0081]

[0082] Among them, λ e is the scale factor. The greater the distance between the feature point and the epipolar line, the greater the dynamic probability.

[0083] Step 6: For the dynamic probabilities under each constraint obtained in Step 4 and Step 5, the proportion of the two in the fusion can be adjusted by adjusting λ m and λ e Finally, the naive Bayes method is used to fuse the two. From the posterior probability formula of naive Bayes:

[0084]

[0085] Among them, c represents the possible categories of the samples, and x represents the samples. From the above formula, for the hypothesis S i = 1 indicates that the state of the i-th feature point is dynamic. The dynamic probability obtained in steps 4 and 5 of the fusion is represented by the following formula:

[0086]

[0087] Among them, η is the prior fusion probability, which represents the dynamic probability of this matching point in the previous frame. If there is no matching point, it is set to 0.5. In this way, the prior information obtained in the previous frame can be fused into the current frame through the set η.

[0088] Step 7: Finally, by setting a threshold, the feature points with a fused dynamic probability greater than the threshold are used as dynamic feature points, and the feature points with a probability less than the threshold are used as static feature points, and the above steps are repeated in the next image until all the image inputs of the camera are processed.

[0089] The above description is only a preferred embodiment of the present invention, and it is not a limitation of the present invention in any other form. Any modification or equivalent change made according to the technical essence of the present invention still belongs to the scope of protection required by the present invention.

Claims

1. A method for removing dynamic feature points of a monocular camera based on lightweight instance segmentation, characterized in that It includes the following steps: Step 1: Use the calibrated camera parameters to correct the distortion of the input camera image, extract the ORB feature points of the image to obtain the ORB feature points of the image, and perform feature point matching with the ORB feature points of the previous frame image; Step 2: While the feature point extraction work in Step 1 is in progress, a new thread is opened to synchronously perform the following work: The image sequence after image distortion correction is input into the trained YOLACT instance segmentation network to perform instance segmentation on the objects with dynamic tendencies including people and vehicles in the input camera image, and the masks of each potential dynamic object are obtained and stored, denoted as , where is the total number of masks in the current frame; Step 3: For each feature point within the potential dynamic object mask obtained in Step 2 and matching the feature points in the previous frame, use the RANSAC algorithm to calculate the homography matrix corresponding to each mask as ; Use the remaining feature points except those within the mask to calculate their pose transformation using the RANSAC algorithm as , set a threshold . When and The difference between is greater than the threshold , it is determined that the object corresponding to the mask is in a moving state. Mark the feature points within the determined dynamic mask as dynamic feature points, which do not participate in the pose estimation to remove their adverse effects on the pose estimation; Among the remaining feature points, there are still dynamic feature points that have not been completely removed. Record the motion states corresponding to each mask for use in subsequent judgments; Step 4: Calculate the dynamic probability of the feature point according to the distance from the feature point to the nearest point of the dynamic mask; Step 5: Calculate the dynamic probability of the feature point under this constraint according to the epipolar constraint; Step 6: Use the Naive Bayes method to fuse the dynamic probabilities calculated in Steps 4 and 5 above to calculate the dynamic probability corresponding to this feature point; Step 7: Finally, by setting a threshold, the feature points with the fused dynamic probability greater than the threshold are used as dynamic feature points, and the feature points with the dynamic probability less than the threshold are used as static feature points, and repeat the above steps in the next image until all the camera image inputs are processed; 2. A method for eliminating dynamic feature points of a monocular camera based on lightweight instance segmentation according to claim 1, characterized in that The RANSAC algorithm in Step 3 is as follows: The first step is to randomly select 4 matching feature points to solve the following equation: ； Among them , are the homogeneous coordinates of the matched feature points, and is the vector of is the homography matrix of . After solving, the homography matrix is obtained; Step 2: Set a threshold, and calculate the value of according to the homography matrix obtained in Step 1. If the calculated value is less than the set threshold, the point is an inlier; otherwise, the point is an outlier. Count the number of corresponding inliers and outliers among the matched feature points; If the calculated value is less than the set threshold, the point is an inlier; otherwise, the point is an outlier. Count the number of corresponding inliers and outliers among the matched feature points; The third step is to compare the number of inliers of the current model with that of the previous best model. If the number of inliers of the current model is more than that of the previous best model, update the best model to the current model and record the corresponding number of inliers; The fourth step is to repeat the above steps 1 to 3 until the iteration ends or the number of inliers corresponding to the current model is more than a certain number; 3. A method for eliminating dynamic feature points of a monocular camera based on lightweight instance segmentation according to claim 1, characterized in that Step 4 is specifically as follows: Calculate the distance from the feature points outside the dynamic mask determined in Step 3 in the image to the nearest edge of the dynamic mask, specifically: ； wherein, are respectively the abscissa and ordinate of the pixel coordinates of the th feature point, and are respectively the abscissa and ordinate of the pixel coordinates of the dynamic mask closest to the feature point; According to the distance from the feature point to the dynamic mask, calculate the dynamic possibility of each feature point according to the following formula: ； Among them, is a proportionality factor of the pixel distance. The smaller the distance between the feature point and the mask, the greater the corresponding dynamic probability.

4. A method for removing dynamic feature points of a monocular camera based on lightweight instance segmentation according to claim 1, characterized in that Step 5 is specifically as follows: According to the epipolar constraint, first calculate the epipolar lines corresponding to the feature points in the previous adjacent frame in the current frame and calculate the distance between the corresponding feature points in the current frame and the epipolar lines, specifically: ； Among them, respectively represent the homogeneous coordinates of the th feature point matched in the current frame and the previous frame, is the fundamental matrix, expressed as: ； where is the camera intrinsic parameter, and are the rotation and translation of the camera relative to the previous frame respectively, represents the skew-symmetric matrix, and is also obtained by the following formula in actual calculations: ； Among them and are the matching feature points in the region where the feature points are located. To calculate the matrix, eight matching points are required; The dynamic probability of the feature point determined by the epipolar constraint is obtained as: ； Among them, is the scale factor. The greater the distance of the feature point from the epipolar line, the greater the dynamic probability.

5. A method for removing dynamic feature points of a monocular camera based on lightweight instance segmentation according to claim 4, characterized in that Step 6 is specifically as follows: For the dynamic probabilities under each constraint obtained in steps 4 and 5, by adjusting and to adjust their proportions in the fusion, and finally using the Naive Bayes method to fuse the two. From the posterior probability formula of Naive Bayes: ； Among them, represents the possible categories of the samples, represents the samples, obtained from the above formula. For the hypothesis represents the status of the j-th feature point is dynamic. The dynamic probabilities obtained in steps 4 and 5 of the fusion are represented by the following formula: ； Among them, is the prior fusion probability, representing the dynamic probability of this matching point in the previous frame. If there is no matching point, it is set to 0.

5. In this way, through the set the prior information obtained in the previous frame is fused into the current frame.

Images