A slam method, device and storage medium with cnn auxiliary feature extraction

Abstract

The application discloses a kind of SLAM method, device and storage medium with CNN auxiliary feature extraction, belong to computer vision technical field, including: obtaining the ORB feature point of current frame of continuous multiple frames RGB image;When the quality of current frame of continuous multiple frames RGB image meets preset condition, then the LoFTR feature point of current frame of RGB image is obtained;ORB feature point and LoFTR feature point are screened and matched.The application obtains the LoFTR feature point of current frame of RGB image when the quality of current frame of continuous multiple frames RGB image meets preset condition, avoids always obtaining LoFTR feature point when the quality of current frame of continuous multiple frames RGB image does not meet preset condition, improves real-time performance while reducing device pressure, matches LoFTR feature point and ORB feature point, solves the problem that traditional feature point method is prone to tracking loss and poor positioning effect in lack of texture, motion blur and other adverse scenes.

Description

TECHNICAL FIELD

[0001] The application relates to a SLAM method, device and storage medium with CNN auxiliary feature extraction, and belongs to the technical field of computer vision. BACKGROUND

[0002] With the development of computer technology, great breakthroughs have been made in automatic driving technology, and the most critical positioning module is obstacle avoidance detection, which is a basic guarantee for path planning. In recent years, SLAM has been more widely applied to automatic driving positioning modules, but for SLAM systems using traditional feature point methods, in poor scenes such as lack of texture and motion blur, tracking loss and poor positioning effect are prone to occur. SUMMARY

[0003] The application aims to overcome the deficiencies in the prior art and provide a SLAM method, device and storage medium with CNN auxiliary feature extraction, which has the effect of generating high-quality matching in low-texture, motion-blur or picture pattern repeated areas.

[0004] To achieve the above-mentioned purpose / solve the above-mentioned technical problems, the application is implemented by using the following technical scheme:

[0005] In a first aspect, a SLAM method with CNN auxiliary feature extraction comprises:

[0006] ORB feature points of a current frame of continuous multiple frames of RGB images are acquired;

[0007] When the quality of the current frame of the continuous multiple frames of RGB images meets a preset condition, LoFTR feature points of the current frame of the RGB images are acquired;

[0008] The ORB feature points and the LoFTR feature points are screened and matched.

[0009] Further, the calculation formula of the quality of the current frame of the RGB images is:

[0010]

[0011] Wherein, err is the re-projection error of the map point to the current frame, n ul is the upper limit value of each frame of feature points, n is the number of current frame feature points after non-maximum suppression, t kf is the time from the last key frame to the current frame, K i is the weight of each index, which can be adjusted according to the actual scene.

[0012] Further, the preset condition of the quality of the current frame of the RGB image is value> preset threshold value; when value> preset threshold value, it is recorded as that the tracking quality of the current frame is poor; and when value≤ preset threshold value, it is recorded as that the tracking quality of the current frame is good.

[0013] Further, the method for obtaining the LoFTR feature point of the current frame of the RGB image comprises:

[0014] obtaining coarse-level features of the current frame image and the previous key frame image;

[0015] matching the coarse-level features of the current frame image with the coarse-level features of the previous key frame image;

[0016] based on the matching result of the coarse-level features, performing fine-level feature matching on the current frame image and the previous key frame image;

[0017] outputting the LoFTR feature point of the current frame of the RGB image.

[0018] Further, the method for matching the coarse-level features of the current frame image with the coarse-level features of the previous key frame image comprises:

[0019] obtaining a matching score matrix between the coarse-level features and calculating a matching probability;

[0020] obtaining an optimal coarse-level feature matching set according to the matching probability.

[0021] Further, the method for performing fine-level feature matching on the current frame image and the previous key frame image comprises:

[0022] obtaining a clipping range of the optimal coarse-level feature;

[0023] obtaining a first matching feature and a second matching feature based on the clipping range;

[0024] calculating a matching probability distribution of the first matching feature and the second matching feature;

[0025] obtaining a matching position on the current frame image according to the matching probability distribution.

[0026] Further, the method for screening and matching the ORB feature points and the LoFTR feature points comprises:

[0027] screening the LoFTR feature points;

[0028] placing the screened LoFTR feature points and the ORB feature points in the same image coordinate system for re-screening to obtain a feature point set;

[0029] matching different types of feature points in the feature point set.

[0030] In a second aspect, a SLAM device with CNN-assisted feature extraction includes:

[0031] A first obtaining module is configured to obtain ORB feature points of a current frame of a plurality of consecutive RGB images;

[0032] A judging module is configured to calculate the quality of the current frame of the RGB image and determine whether the quality of the current frame of the RGB image meets a preset condition;

[0033] A second obtaining module is configured to obtain LoFTR feature points of the current frame of the RGB image;

[0034] A fusion screening module is configured to screen and match the ORB feature points and the LoFTR feature points.

[0035] In a third aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above method.

[0036] In a fourth aspect, a computing device includes one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0037] Compared with the prior art, the present application has the following beneficial effects:

[0038] In the present application, the LoFTR feature points of the current frame of the RGB image are obtained only when the quality of the current frame of the plurality of consecutive RGB images meets the preset condition, thereby avoiding the continuous obtaining of the LoFTR feature points when the quality of the current frame of the plurality of consecutive RGB images does not meet the preset condition, reducing the pressure on the device, improving the real-time performance, and combining the LoFTR feature points and the ORB feature points for matching, thereby solving the problem of tracking loss and poor positioning effect of the traditional feature point method in poor scenes such as lack of texture and motion blur. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a flowchart of a SLAM method with CNN-assisted feature extraction provided by an embodiment of the present application;

[0040] Figure 2 is a quadtree screening process schematic diagram of a SLAM method with CNN-assisted feature extraction provided by an embodiment of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application.

[0042] Embodiment 1

[0043] As shown in Figure 1 , a SLAM method with CNN auxiliary feature extraction comprises:

[0044] ORB feature points of a current frame of continuous multiple frames of RGB images are acquired;

[0045] It is judged whether the quality of the current frame of the continuous multiple frames of RGB images meets a preset condition, and specifically, the calculation formula of the quality of the current frame of the RGB images is:

[0046]

[0047] Wherein, err is the re-projection error of the map point to the current frame, n ul is the upper limit value of each frame of feature points, n is the number of current frame feature points after non-maximum suppression, t kf is the time from the last key frame to the current frame, K i is the weight of each index, which can be adjusted according to the actual scene; when value> preset threshold, it is recorded as the current frame tracking quality is poor; when value≤ preset threshold, it is recorded as the current frame tracking quality is good.

[0048] When the quality of the current frame of the continuous multiple frames of RGB images meets the preset condition: the preset condition of the quality of the current frame of the RGB images is value> preset threshold, that is, the current frame tracking quality is poor, then the LoFTR feature points of the current frame of the RGB images are acquired;

[0049] Specifically, coarse-level features of the current frame image and the last key frame image are acquired.

[0050] The coarse-level features of the current frame image and the coarse-level features of the last key frame image are matched, and a matching score matrix S between each coarse-level feature is acquired.

[0051]

[0052] The matching probability P C is calculated as:

[0053] P C (i, j) = softmax(S(i, ·))j·softmax(S(·, j));

[0054] Then, some outlier matching pairs are filtered out through the nearest neighbor algorithm MNN to obtain a rough matching prediction Mc.

[0055] According to the matching probability P CObtain the optimal coarse-level feature matching set (i, j), wherein i and j represent the point pair corresponding to the current frame image and the previous frame key frame image.

[0056] Based on the coarse-level feature matching result, fine-level feature matching is performed on the current frame image and the previous frame key frame image; specifically:

[0057] Obtain the clipping range of the optimal coarse-level feature, and clip in the local window with a size of WxW with i and j as the center respectively;

[0058] Based on the clipping range, obtain the first matching feature FA and the second matching feature FB;

[0059] Calculate the matching probability of the center feature of the first matching feature FA and all features in the second matching feature FB, and then calculate the probability distribution, so as to obtain the final matching position of the sub-pixel coordinates on the current frame image.

[0060] In the coarse-level feature matching process, a 256-dimensional vector is generated as the descriptor of each coarse-level feature; in the fine-level feature matching process, a 128-dimensional vector is generated as the descriptor of each fine-level feature; finally, the 256-dimensional vector extracted in the coarse-level feature matching process and the 128-dimensional vector extracted in the fine-level feature matching process are spliced to obtain a 384-dimensional vector as the LoFTR feature point.

[0061] The ORB feature points and the LoFTR feature points are screened and matched, specifically, the LoFTR feature points are screened, first, the canny edge detection algorithm is used to detect the edge information in the image, and it is converted into a binary image; then, the Hough transform is used to detect the straight lines in the image, and a binary image is generated as a feature mask; wherein, only the pixel points on the straight lines are marked as available, because the feature points on some structures in the environment are more diverse and more easily matched accurately.

[0062] As shown in Figure 2 , the screened LoFTR feature points and the ORB feature points are placed in the same image coordinate system and screened again to obtain a feature point set; specifically, the current frame image is screened by a quadtree, wherein for each node, the strategy for retaining feature points is as follows: first, retain the ORB feature points extracted using a high response threshold, if there are no ORB feature points, retain the LoFTR feature points in the mask, if there are no LoFTR feature points, finally retain the ORB feature points extracted using a low response threshold.

[0063] Match according to different types of feature points in the feature point set, specifically:

[0064] For LoFTR feature points in the feature point set, brute force matching is performed; a KNN algorithm is used to calculate the Euclidean distance between LoFTR feature point descriptors, and then feature point matching between the current frame and the previous key frame is realized;

[0065] For ORB feature points in the feature point set, first, by assuming constant velocity, the key points of the previous key frame are projected to the current frame, and matching items are searched in the local window, if failed, then try to use bag of words (BoW) to find matching items between the current frame and the previous key frame.

[0066] Embodiment 2

[0067] A SLAM device with CNN auxiliary feature extraction, comprising:

[0068] A first acquisition module is configured to acquire ORB feature points of a current frame of continuous multiple frames of RGB images.

[0069] A judgment module is configured to calculate the quality of the current frame of RGB images and determine whether the quality of the current frame of RGB images meets a preset condition.

[0070] A second acquisition module is configured to acquire LoFTR feature points of the current frame of RGB images.

[0071] A fusion screening module is configured to screen and match the ORB feature points and the LoFTR feature points.

[0072] Embodiment 3

[0073] A computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the following method:

[0074] ORB feature points of a current frame of continuous multiple frames of RGB images are acquired.

[0075] When the quality of the current frame of continuous multiple frames of RGB images meets a preset condition, LoFTR feature points of the current frame of RGB images are acquired.

[0076] ORB feature points and LoFTR feature points are screened and matched.

[0077] Embodiment 4

[0078] A computing device, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise instructions for executing the following method:

[0079] ORB feature points of a current frame of continuous multiple frames of RGB images are acquired.

[0080] When the quality of the current frame of the plurality of consecutive RGB images meets a preset condition, a LoFTR feature point of the current frame of the RGB image is acquired.

[0081] The ORB feature points and the LoFTR feature points are screened and matched.

[0082] The above only describes the preferred embodiments of the present application, and it should be noted that those skilled in the art can make several improvements and modifications without departing from the technical principles of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A SLAM method with CNN-aided feature extraction, characterized in that, The method comprises the following steps: obtain ORB feature points of a current frame of continuous multiple frames of RGB images; when the quality of the current frame of RGB images meets a preset condition, obtain LoFTR feature points of the current frame of RGB images; the method for obtaining the LoFTR feature points of the current frame of RGB images comprises the following steps: obtain coarse-level features of the current frame of images and a previous key frame of images; match the coarse-level features of the current frame of images with the coarse-level features of the previous key frame of images, which comprises the following steps: obtain a matching score matrix between the coarse-level features and calculate a matching probability; obtain an optimal coarse-level feature matching set according to the matching probability; based on the coarse-level feature matching result, perform fine-level feature matching on the current frame of images and the previous key frame of images; output the LoFTR feature points of the current frame of RGB images; the calculation formula of the quality of the current frame of RGB images is: ； wherein err is the re-projection error of the map point to the current frame, is an upper limit value of the feature points of each frame, n is the number of the feature points of the current frame after non-maximum suppression, is the time from the last key frame to the current frame, is the weight of each index, and the weights of the indexes are adjusted according to the actual scene. screen and match the ORB feature points and the LoFTR feature points, which comprises the following steps: screen the LoFTR feature points; place the screened LoFTR feature points and the ORB feature points in the same image coordinate system to screen again, and obtain a feature point set; specifically, perform quadtree screening on the current frame of images, wherein for each node, the strategy for retaining feature points is as follows: preferentially retain the ORB feature points extracted by using a high-response threshold, retain the LoFTR feature points within a mask if there is no ORB feature point, and retain the ORB feature points extracted by using a low-response threshold last if there is no LoFTR feature point; match different types of feature points in the feature point set, specifically: perform brute-force matching on the LoFTR feature points in the feature point set; use a KNN algorithm to calculate the Euclidean distance between LoFTR feature point descriptors, and thus realize feature point matching between the current frame and the previous key frame; for the ORB feature points in the feature point set, first, project the key points of the previous key frame to the current frame by assuming that the velocity is constant, and search for matching items in a local window, if the search fails, try to use a bag-of-words (BoW) to find matching items between the current frame and the previous key frame.

2. The SLAM method with CNN-aided feature extraction of claim 1, wherein, the preset condition of the quality of the current frame of RGB images is value > a preset threshold; when value > the preset threshold, it is recorded as poor tracking quality of the current frame; when value ≤ the preset threshold, it is recorded as good tracking quality of the current frame. 3.The SLAM method with CNN-aided feature extraction of claim 1, wherein, the method for performing fine-level feature matching on the current frame of images and the previous key frame of images comprises the following steps: obtain a clipping range of the optimal coarse-level feature; obtain first matching features and second matching features based on the clipping range; calculate a matching probability distribution of the first matching features and the second matching features; obtain a matching position on the current frame of images according to the matching probability distribution.

4. A SLAM apparatus with CNN-aided feature extraction, characterized by, The method comprises the following steps: a first obtaining module is configured to obtain ORB feature points of a current frame of continuous multiple frames of RGB images; a second obtaining module is configured to obtain LoFTR feature points of the current frame of RGB images when the quality of the current frame of continuous multiple frames of RGB images meets a preset condition; the method for obtaining the LoFTR feature points of the current frame of RGB images comprises the following steps: obtain coarse-level features of the current frame of images and a previous key frame of images; Matching the coarse-level features of the current frame image with the coarse-level features of the previous key frame image, comprising: Obtaining a matching score matrix between the coarse-level features and calculating a matching probability; Obtaining an optimal coarse-level feature matching set according to the matching probability; Based on the coarse-level feature matching result, performing fine-level feature matching on the current frame image and the previous key frame image; Outputting the LoFTR feature points of the current frame of the RGB image; The calculation formula of the quality of the current frame of the RGB image is: ； wherein err is the re-projection error of the map point to the current frame, is an upper limit value of the feature points of each frame, n is the number of the feature points of the current frame after non-maximum suppression, is the time from the last key frame to the current frame, is the weight of each index, and the weights of the indexes are adjusted according to the actual scene. The fusion screening module is used for screening and matching the ORB feature points and the LoFTR feature points, comprising: Screening the LoFTR feature points; The screened LoFTR feature points and the ORB feature points are placed in the same image coordinate system for re-screening to obtain a feature point set; specifically, the current frame image is subjected to quadtree screening, wherein for each node, the strategy for retaining feature points is as follows: the ORB feature points extracted using a high response threshold are preferentially retained, if there are no ORB feature points, the LoFTR feature points within the mask are retained, if there are no LoFTR feature points, the ORB feature points extracted using a low response threshold are finally retained; different types of feature points in the feature point set are matched, specifically: for the LoFTR feature points in the feature point set, brute force matching is performed; the KNN algorithm is used to calculate the Euclidean distance between the LoFTR feature point descriptors, thereby realizing the feature point matching between the current frame and the previous key frame; For the ORB feature points in the feature point set, first, by assuming constant velocity, the key points of the previous key frame are projected to the current frame, and matching items are searched in the local window, if failed, the bag of words BoW is used to find matching items between the current frame and the previous key frame.

5. A computer readable storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to realize the steps of the method of any one of claims 1-3.

6. A computing device, comprising: Comprising: One or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any one of the methods according to claims 1-3.

Images