A grayscale histogram-based ORB-SLAM3 loop detection acceleration method

Abstract

The present invention discloses an ORB-SLAM3 loop detection acceleration method based on grayscale histogram, step S1: when a local mapping thread inserts a key frame into a loop detection thread, the grayscale histogram of the key frame is extracted, and its standard deviation, mean and other information are calculated; step S2: high-quality loop candidate key frames are screened out at the front end of the loop detection; step S3: loop feasibility verification is performed on the loop candidate key frames; step S4: loop correction is performed at the back end of the loop detection. The technical solution of the present invention is adopted, and the input key frames are pre-screened by using the image grayscale histogram information with light calculation, so as to improve the quality of the input key frames for loop verification, reduce the number of invalid calculations caused by low key frame quality, and improve the real-time performance of the ORB-SLAM3 system.

Description

Technical Field

[0001] The present invention relates to the field of robot synchronous positioning and map creation, and in particular to an ORB-SLAM3 loop detection acceleration method based on grayscale histogram. Background Art

[0002] In the past decade, the development of simultaneous localization and mapping (SLAM) has been very rapid, and the focus of research has gradually shifted from the initial lidar SLAM to visual SLAM. The real-time performance of the SLAM system is considered to be one of the key issues in current research, which is of great significance to the practical application and application expansion of robots. Among the many SLAM frameworks currently available, ORB-SLAM3 is one of the best open source frameworks. It uses the bag-of-words model for long-term data association, can achieve loop closure of the same map and fusion of multiple maps, and uses the method of combining geometric verification with temporal verification to improve the recall rate. However, due to the low quality of the loop candidate keyframes generated by this method, a large amount of invalid calculations are performed in the loop detection process, which has a huge impact on the real-time performance of the SLAM system.

[0003] The loop detection method in the prior art mainly has the following two problems:

[0004] 1. The quality of the loop closure candidate key frames is poor, resulting in multiple invalid calculations and even missing the optimal time for loop closure.

[0005] 2. The loop detection thread has a large amount of calculation, and as the system running time increases, the time consumption will also increase, resulting in a decrease in the real-time performance of the SLAM system.

[0006] Therefore, in view of the defects of the prior art, it is necessary to propose a technical solution to solve the technical problems existing in the prior art. Summary of the invention

[0007] In view of this, it is necessary to provide an ORB-SLAM3 loop detection acceleration method based on grayscale histogram, extract the grayscale histogram and its standard deviation and mean information from the grayscale image of the input key frame, and then use this information to pre-screen the candidate key frames for loop closure, and then perform loop verification and loop correction, thereby greatly reducing the amount of calculation of the loop detection thread for each frame, while improving the quality of the candidate key frames, which not only improves the real-time performance but also improves the accuracy.

[0008] In order to overcome the defects of the prior art, the technical solution of the present invention is as follows:

[0009] An ORB-SLAM3 loop detection acceleration method based on grayscale histogram includes the following steps:

[0010] Step S1: When the local mapping thread inserts a key frame into the loop detection thread, the grayscale histogram of the key frame is extracted and its standard deviation, mean and other information are calculated;

[0011] Step S2: Screening out high-quality loop candidate key frames at the front end of loop detection;

[0012] Step S3: Verify the feasibility of loop closure on the loop closure candidate key frames;

[0013] Step S4: performing loop correction at the back end of loop detection; wherein step S1 further includes:

[0014] Step S11: Adaptively crop the key frame grayscale image inserted into the loop detection thread according to I′ gray =Rect(I gray ,θ 1 ,λ 1 ,θ 2 ,λ 2 ), where I′ gray is the cropped grayscale image, indicating that I gray From the coordinates (θ 1 ,λ 1 ) is intercepted to (θ 2 ,λ 2 ), the specific coordinates are automatically adjusted according to the input image size;

[0015] Step S12: Calculate I′ gray The grayscale histogram Hist has a total of 256 grayscale levels ranging from 0 to 255;

[0016] Step S13: According to Normalize the range of Hist to get H, where x in 、x out They are the original values ​​corresponding to each gray level and the normalized values ​​corresponding to each gray level, and the value range is mapped from (γ, δ) to (α, β);

[0017] Step S14: Calculate the mean and standard deviation of the grayscale histogram after range normalization. The calculation formula is as follows:

[0018]

[0019]

[0020] Where M×N is the size of the grayscale histogram H, and P(i,j) represents the value of the i-th row and j-th column of the histogram;

[0021] Step S15: Establish a two-dimensional container to store the frame data structure, use the mean as the horizontal coordinate and the standard deviation as the vertical coordinate, and put the key frame into the corresponding position of the container according to (μ, SD);

[0022] Step S2 selects high-quality loop candidate key frames at the front end of loop detection, and this step further includes:

[0023] Step S21: according to the mean and standard deviation of the current key frame, a key frame group with similar mean and standard deviation to the current frame is taken out from the two-dimensional container according to (μ, SD), that is, all key frames with similar positions are stored in the two-dimensional container;

[0024] Step S22: traverse all word bag nodes of the current key frame, and extract all key frame KFs that have common nodes with the current key frame from the above key frame group according to the inverted index of the word bag;

[0025] Step S23: obtaining the number m of the common view key frames of the current frame, and directly removing from the KFs all key frames whose ID difference with the current frame is less than m, to obtain new KFs;

[0026] Step S24: Find the best three key frames from the above KFs as a candidate key frame group; step S24 further includes:

[0027] Step S241: Calculate the similarity scores of the two frames using the bag-of-words vector;

[0028] Step S242: The key frames with similarity scores lower than the threshold are marked and will not be detected with the current frame in the future. For the key frames with similarity scores higher than the threshold, the scores of their adjacent key frames are calculated;

[0029] Step S243: Compare the scores of the key frames with similarity scores higher than the threshold with those of the adjacent key frames, and return the three best key frames with scores much higher than those of the adjacent key frames as the candidate key frame group;

[0030] Step S25: Calculate the similarity between the key frames in the candidate key frame group and the grayscale histogram of the current frame, and remove the candidate key frames with similarity less than 0.85. The similarity calculation formula is as follows;

[0031]

[0032]

[0033] The grayscale histograms of the current frame and the candidate key frame after processing are H 1 and H 2 , H 1 and H 2The similarity between them is d(H 1 ,H 2 ), d(H 1 ,H 2 ) is closer to 1, the higher the similarity between the two frames is considered;

[0034] Step S3 verifies the feasibility of the loop closure candidate, and this step further includes:

[0035] Step S31: Geometry verification. As long as 3 of the 5 common-view keyframes of the current keyframe meet the conditions (match the candidate keyframe group successfully), the geometry verification is considered to have passed. If there are less than 3, proceed to S32;

[0036] Step S32: Timing verification. If the key frame input subsequently can also be successfully matched with the candidate key frame group, the timing verification is considered to have passed. If the sum of the number of successful geometric verifications in the previous step is 3, the preliminary verification of loop feasibility is considered to have passed. If the timing verification of two consecutive frames fails, the timing verification is considered to have failed;

[0037] Step S33: Finally, the transformation values ​​of the roll angle, pitch angle, and yaw angle are verified. Only when all three values ​​are less than a certain threshold value, the loop feasibility verification is considered to have finally passed.

[0038] Step S4 performs loop correction at the back end of loop detection, and this step further includes:

[0039] Step S41: Calculate the Sim3 transformation of the key frame;

[0040] Step S42: posture propagation;

[0041] Step S43: map point correction.

[0042] Compared with the prior art, the technical solution of the present invention can greatly improve the real-time performance. The present invention uses the information such as the grayscale histogram of the key frame to screen at the loop detection front end, and further eliminates the loop candidate frame according to the similarity between the loop candidate frame and the current key frame. The process is extremely lightweight and can effectively remove a large number of invalid calculations. And by using the similarity between the grayscale histograms, a very high similarity can be guaranteed at the grayscale distribution level of the entire image, thereby improving the quality of the loop candidate frame, making the loop candidate frame and the current frame very similar, thereby reducing the number of calculations during the loop, and greatly improving the real-time performance of the SLAM system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A framework flow chart of an ORB-SLAM3 loop detection acceleration method based on grayscale histogram provided by the present invention;

[0044] Figure 2 A detailed flow chart of step S24 in an ORB-SLAM3 loop detection acceleration method based on grayscale histogram provided by the present invention;

[0045] Figure 3 This is a time comparison diagram of the loop detection process of the present invention and the ORB-SLAM3 algorithm on the Euroc dataset;

[0046] Figure 4 This is a comparison chart of loop detection time for each key frame of the present invention and the ORB-SLAM3 algorithm on the V203 sequence;

[0047] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0048] The technical solution provided by the present invention will be further described below in conjunction with the accompanying drawings.

[0049] In the current visual SLAM systems with relatively good performance, most of them use the bag-of-words model for closed-loop detection, so as to achieve long-term data association and error elimination process. The bag-of-words model has less computational effort and can make the closed-loop process have a higher recall rate. However, the effectiveness of the bag-of-words model is deeply affected by the training set. A small number of images cannot train an effective bag-of-words model, and the bag-of-words model is not completely accurate. It is easy to have false positive data. In addition, the loop detection method based on the bag-of-words only cares about the presence or absence of words, not the order of words, which can easily cause perceptual bias, so it needs to be verified by other methods. The grayscale histogram of the image has translation, rotation, and scaling invariance, and its various indicators can be quickly calculated and compared, which can reflect the overall pixel distribution law of an image at a relatively low computational cost, and is very suitable for use in SLAM systems.

[0050] See also Figure 1 , Figure 2 The present invention provides an ORB-SLAM3 loop detection acceleration method based on grayscale histogram. Figure 1 The figure shows the architecture diagram of the ORB-SLAM3 loop detection acceleration method based on grayscale histogram of the present invention. Generally speaking, the present invention includes four major steps. Step S1: when the local mapping thread inserts a key frame into the loop detection thread, the grayscale histogram of the key frame is extracted, and its standard deviation, mean and other information are calculated; Step S2: high-quality loop candidate key frames are screened out at the front end of the loop detection; Step S3: loop feasibility verification is performed on the loop candidate key frames; Step S4: loop correction is performed at the back end of the loop detection;

[0051] When the local mapping thread inserts a key frame into the loop detection thread in step S1, the gray-scale histogram of the key frame is extracted, and information such as its standard deviation and mean is calculated. The specific steps are as follows:

[0052] Step S11: Adaptive cropping is performed on the grayscale image of the key frame inserted into the loop detection thread. According to I′ gray =Rect(I gray ,θ 1 ,λ 1 ,θ 2 ,λ 2 ), where I′ gray is the cropped grayscale image, indicating that I gray is intercepted from the coordinates (θ 1 ,λ 1 ) to (θ 2 ,λ 2 ). The specific coordinates are automatically adjusted according to the size of the input image;

[0053] Step S12: Calculate the gray-scale histogram Hist of I′ gray , taking a total of 256 gray levels from 0 to 255;

[0054] Step S13: Perform range normalization on Hist according to to obtain H, where x in , x out are the values corresponding to each original gray level and the values corresponding to each gray level after normalization, respectively. The value range is mapped from (γ,δ) to (α,β);

[0055] Step S14: Calculate the mean and standard deviation of the range-normalized gray-scale histogram. The calculation formulas are as follows:

[0056]

[0057]

[0058] where M×N is the size of the gray-scale histogram H, and P(i,j) represents the value at the i-th row and j-th column of the histogram;

[0059] Step S15: Establish a two-dimensional container to store the data structure of the frame. Using the mean as the abscissa and the standard deviation as the ordinate, place the key frame at the corresponding position in the container according to (μ,SD);

[0060] In step S2, high-quality loop candidate key frames are screened out at the front end of the loop detection. The specific steps are as follows:

[0061] Step S21: according to the mean and standard deviation of the current key frame, a key frame group with similar mean and standard deviation to the current frame is taken out from the two-dimensional container according to (μ, SD), that is, all key frames with similar positions are stored in the two-dimensional container;

[0062] Step S22: traverse all word bag nodes of the current key frame, and extract all key frame KFs that have common nodes with the current key frame from the above key frame group according to the inverted index of the word bag;

[0063] Step S23: obtaining the number m of the common view key frames of the current frame, and directly removing from the KFs all key frames whose ID difference with the current frame is less than m, to obtain new KFs;

[0064] Step S24: Find the best three key frames from the above KFs as candidate key frame groups;

[0065] Figure 2 The figure shows a detailed flow chart of the above steps in an ORB-SLAM3 loop detection acceleration method based on grayscale histogram provided by the present invention, which further includes the following steps:

[0066] Step S241: Calculate the similarity scores of the two frames using the bag-of-words vector;

[0067] Step S242: The key frames with similarity scores lower than the threshold are marked and will not be detected with the current frame in the future. For the key frames with similarity scores higher than the threshold, the scores of their adjacent key frames are calculated;

[0068] Step S243: Compare the scores of the key frames with similarity scores higher than the threshold with those of the adjacent key frames, and return the three best key frames with scores much higher than those of the adjacent key frames as the candidate key frame group;

[0069] Step S25: Calculate the similarity between the key frames in the candidate key frame group and the grayscale histogram of the current frame, and remove the candidate key frames with similarity less than 0.85. The similarity calculation formula is as follows;

[0070]

[0071]

[0072] The grayscale histograms of the current frame and the candidate key frame after processing are H 1 and H 2 , H 1 and H 2 The similarity between them is d(H 1 ,H 2 ), d(H 1 ,H 2) is closer to 1, the higher the similarity between the two frames is considered;

[0073] Step S3 verifies the feasibility of looping on the selected frames after looping, which specifically includes the following steps:

[0074] Step S31: Geometry verification. As long as 3 of the 5 common-view keyframes of the current keyframe meet the conditions (match the candidate keyframe group successfully), the geometry verification is considered to have passed. If there are less than 3, proceed to S32;

[0075] Step S32: Timing verification. If the key frame input subsequently can also be successfully matched with the candidate key frame group, the timing verification is considered to have passed. If the sum of the number of successful geometric verifications in the previous step is 3, the preliminary verification of loop feasibility is considered to have passed. If the timing verification of two consecutive frames fails, the timing verification is considered to have failed;

[0076] Step S33: Finally, the transformation values ​​of the roll angle, pitch angle, and yaw angle are verified. Only when all three values ​​are less than a certain threshold value, the loop feasibility verification is considered to have finally passed.

[0077] Step S4 performs loop correction at the back end of loop detection, which specifically includes the following steps:

[0078] Step S41: Calculate the Sim3 transformation of the key frame;

[0079] Step S42: posture propagation;

[0080] Step S43: map point correction.

[0081] The input data used in the present invention adopts the images and IMU data in the Euroc dataset. The Euroc dataset is a visual inertial dataset collected by a micro-aircraft, which contains 11 binocular video sequences. The dataset scenes are indoor rooms and factories, among which V103, V202 and V203 are sequences with loops. The simulation experiment platform of this paper is a computer with Intel(R) Core(TM) i7-7700HQ CPU@2.80GHz and 24G memory. All video sequences in the dataset are applied to this experiment to observe the relationship between the loop detection time after being processed by the algorithm of this paper and the loop detection time of the unprocessed ORB-SLAM3.

[0082] In order to verify the technical effect of the present invention, the effectiveness of the present invention will be verified by algorithm comparison:

[0083] In the comparison between this method and the original method of ORB-SLAM3, both are run ten times and the median is taken as the final comparison result. The time consumption of the algorithm in the loop detection process is used as a measurement indicator for comparison, and the time efficiency improved by this method compared with the original algorithm is calculated. Secondly, since the V203 sequence is a fast-moving sequence with motion blur, a large number of repeated scenes begin to appear in the middle and late stages of the V203 sequence. The SLAM system should complete the loop detection and loop correction process in this scene, but not all frames meet the loop conditions during this period. There are many frames with blur and excessive angle changes in the rpy direction. Although similar feature points can be extracted and matched, they will be discarded in the end. Therefore, there are several cases in the V203 sequence where a large amount of time is invalidly calculated due to the above reasons, and each calculation consumes tens of milliseconds, which seriously wastes computer resources. Therefore, taking V203 as an example, the time consumed by each key frame in the sequence in the loop detection process is calculated, and a large amount of time consumption during the loop is intuitively displayed, and the improvement of this algorithm is demonstrated.

[0084] See Table 1. Figure 3 , Figure 4 As shown, the experimental data results of the present invention and the original ORB-SLAM3 algorithm are compared. Table 1 shows the loop detection time consumption data of the present invention and the original ORB-SLAM3 algorithm and the improvement of time efficiency; Figure 3 This is a comparison chart of the average loop detection time of the present invention and the original ORB-SLAM3 algorithm on 11 sequences of the Euroc dataset; Figure 4 This is a comparison chart of the loop detection time of each key frame in the V203 sequence between the present invention and the original ORB-SLAM3 algorithm.

[0085] Table 1 Comparison of loop detection time between the algorithm of the present invention and the original algorithm of ORB-SLAM3 and time efficiency improvement (ms)

[0086]

[0087] Table 1 above lists the comparison of loop detection time between the algorithm of the present invention and the original ORB-SLAM3 algorithm, as well as the percentage of time efficiency improvement. Figure 3 The average detection time of each key frame in 11 sequences is shown, which can vividly show the time consumption reduction of this method compared with ORB-SLAM3. Figure 4The loop detection time consumption of each key frame in the V203 sequence is shown, which vividly reflects the time consumption of each frame in the whole sequence, especially the time consumption when calculating the loop is particularly large. Since it is better reflected in the figure, the time is taken logarithm. In summary, the present invention improves the loop detection thread of ORB-SLAM3 by utilizing the grayscale histogram, greatly reduces the time consumption of each frame in the loop detection process, and improves the real-time performance of the SLAM system, wherein the real-time performance improvement is more obvious when there is a loop or the running time is long. As shown in Table 1, in all sequences of the Euroc data set, the time efficiency of the present invention is improved by an average of about 55.96% compared with the original method of ORB-SLAM3.

[0088] The above embodiments are only used to help understand the method and core idea of ​​the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0089] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A grayscale histogram-based ORB-SLAM3 loop detection acceleration method, It is characterized in that At least the following steps are included: Step S1: When the local mapping thread inserts a key frame into the loop detection thread, the grayscale histogram of the key frame is extracted and its standard deviation, mean and other information are calculated; Step S2: Screening out high-quality loop candidate key frames at the front end of loop detection; Step S3: Verify the feasibility of loop closure on the loop closure candidate key frames; Step S4: performing loop correction at the back end of loop detection; Wherein, the step S1 further comprises: Step S11: Adaptively crop the key frame grayscale image inserted into the loop detection thread according to I′ gray =Rect(I gray ,θ 1 ,λ 1 ,θ 2 ,λ 2 ), where I′ gray is the cropped grayscale image, indicating that I gray From the coordinates (θ 1 ,λ 1 ) is intercepted to (θ 2 ,λ 2 ), the specific coordinates are automatically adjusted according to the input image size; Step S12: Calculate I′ gray The grayscale histogram Hist has a total of 256 grayscale levels ranging from 0 to 255; Step S13: According to Normalize the range of Hist to get H, where x in 、x out They are the original values ​​corresponding to each gray level and the normalized values ​​corresponding to each gray level, and the value range is mapped from (γ, δ) to (α, β); Step S14: Calculate the mean and standard deviation of the grayscale histogram after range normalization. The calculation formula is as follows: Where M×N is the size of the grayscale histogram H, and P(i, j) represents the value of the i-th row and j-th column of the histogram; Step S15: Establish a two-dimensional container to store the frame data structure, use the mean as the horizontal coordinate and the standard deviation as the vertical coordinate, and put the key frame into the corresponding position of the container according to (μ, SD); The step S2 further comprises: Step S21: according to the mean and standard deviation of the current key frame, a key frame group with similar mean and standard deviation to the current frame is taken out from the two-dimensional container according to (μ, SD), that is, all key frames with similar positions are stored in the two-dimensional container; Step S22: traverse all word bag nodes of the current key frame, and extract all key frame KFs that have common nodes with the current key frame from the above key frame group according to the inverted index of the word bag; Step S23: obtaining the number m of the common view key frames of the current frame, and directly removing from the KFs all key frames whose ID difference with the current frame is less than m, to obtain new KFs; Step S24: Find the best three key frames from the above KFs as candidate key frame groups; Step S25: Calculate the similarity between the key frames in the candidate key frame group and the grayscale histogram of the current frame, and remove the candidate key frames with similarity less than 0.

85. The similarity calculation formula is as follows; The grayscale histograms of the current frame and the candidate key frame after processing are H 1 and H 2 , H 1 and H 2 The similarity between them is d(H 1 , H 2 ), d(H 1 , H 2 ) is closer to 1, the higher the similarity between the two frames is considered; The step S24 further comprises: Step S241: Calculate the similarity scores of the two frames using the bag-of-words vector; Step S242: key frames with similarity scores lower than the threshold are marked and will not be detected with the current frame in the future; key frames with similarity scores higher than the threshold are scored by calculating the scores of their adjacent key frames; Step S243: Compare the scores of the key frames with similarity scores higher than the threshold with their adjacent key frames, and return the three best key frames with scores much higher than the scores of the adjacent key frames as the candidate key frame group; The step S3 further comprises: Step S31: geometry verification; as long as 3 of the 5 common view key frames of the current key frame meet the conditions, the match with the candidate key frame group is successful, and the geometry verification is considered to be passed; if there are less than 3, proceed to S32; Step S32: timing verification; if the key frame input subsequently can also be successfully matched with the candidate key frame group, the timing verification is considered to have passed, and if the sum of the number of successful geometric verifications in the previous step is 3, the preliminary verification of loop feasibility is considered to have passed; if the timing verification of two consecutive frames fails, the timing verification is considered to have failed; Step S33: Finally, the transformation values ​​of the roll angle, pitch angle, and yaw angle are verified. Only when all three values ​​are less than a certain threshold value, the loop feasibility verification is considered to have finally passed. The step S4 further comprises: Step S41: Calculate the Sim3 transformation of the key frame; Step S42: posture propagation; Step S43: map point correction.

Images