An improved visual slam loop detection method based on image block region feature points

Abstract

The present application belongs to image processing, and particularly to visual SLAM technology, and discloses an improved method for visual SLAM loop detection based on image block region feature points. The method first converts the input image into a gray image and divides the image into blocks through a grid, calculates the gradient direction histogram of the feature points in the blocks, and obtains the gradient vectors of the feature points; judges the weight of the gradient vectors of the feature points in the blocks, and regards the blocks whose weight does not reach a set threshold as invalid blocks and the blocks whose weight reaches the set threshold as valid blocks; constructs a vocabulary tree model for the valid blocks in the current frame and the gradient vectors of the corresponding feature points, and iterates and updates the content information of the vocabulary tree continuously. The similarity between two images is measured by searching the updated vocabulary tree model, so that better image retrieval and matching results are obtained, and more accurate loop detection is realized. In this way, the problems of error accumulation and camera trajectory drift caused by long-time positioning and three-dimensional map construction of the camera are overcome.

Description

Technical Field

[0001] This invention belongs to the field of image processing and visual simultaneous localization and mapping (SLAM) technology, and particularly relates to an improved visual SLAM loop closure detection method based on feature points of image block regions. Technical Background

[0002] During long-term operation, visual SLAM systems are susceptible to environmental noise, leading to the accumulation of errors in the visual sensors. This accumulation of errors ultimately results in severe distortion of the localization and mapping results. To address this critical issue, a loop closure detection module is incorporated into the visual SLAM framework. This module determines whether the camera has returned to its original position while acquiring environmental information. Therefore, an accurate loop closure detection module provides a reliable basis for the robustness of the visual SLAM system and is crucial for optimizing the overall camera trajectory and the map.

[0003] In visual SLAM systems, the loop closure detection module plays a crucial role in camera localization and map building. Therefore, based on their implementation principles, loop closure detection methods can be broadly categorized into geometric information-based loop closure algorithms and appearance information-based loop closure methods. Appearance information-based loop closure detection methods primarily rely on image matching, utilizing color and texture variations in the image, and are easily affected by ambient lighting. Geometric information-based loop closure detection algorithms, on the other hand, determine whether the current position is near a previous historical position. This principle is simple and easy to implement. Furthermore, due to its rotation invariance, it can overcome noise generated by the camera itself when the camera rotates. Additionally, based on geometric feature information, feature point-based loop closure detection can extract key points and descriptors from the image to aid in data association.

[0004] Feature point methods utilizing geometric information primarily enrich the local image by extracting texture information from the input image, thereby establishing data association between image frames and calculating the camera pose and corresponding 3D point coordinates. Geometric feature point methods are less affected by changes in camera pose. To retain the advantages of descriptors, the input image information is divided into image blocks, and point feature extraction and description are performed in regions with obvious image block features. However, common global feature point descriptor extraction and matching are computationally intensive and easily affected by noise variations. Therefore, addressing the problem of camera pose error accumulation through image block information is of great significance. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes an image block region feature point extraction method that calculates gradient direction histograms by dividing the image into blocks and uses weights to determine valid image blocks. Loop closure detection is then performed using valid image blocks, which avoids wasting a lot of time matching weak texture regions with indistinct features.

[0006] Therefore, to overcome the error accumulation and camera trajectory drift problems caused by long-term camera positioning and 3D map construction, the method of this invention first converts the input image into a grayscale image and divides the image into blocks using a grid. Simultaneously, it calculates the gradient direction histogram of feature points in each image block to obtain the gradient vector of the feature points. Then, it judges the gradient vector weights of the feature points in the image block; if the weights do not reach a set threshold, the image block is considered invalid; if the weights reach the set threshold, it is considered valid. Finally, it constructs a vocabulary tree model using the valid image blocks in the current frame and the gradient vectors of the corresponding feature points, continuously iterating and updating the content information of the vocabulary tree. When performing the loop closure detection stage, it searches and measures the similarity between two frames of images in the updated vocabulary tree model, thereby obtaining better image retrieval and matching results and achieving more accurate loop closure detection. This completes the invention. Its overall technical solution involves dividing the image into blocks using a grid, calculating the gradient histogram of each image block, extracting features from the valid image blocks of the current frame using the gradient direction histogram, constructing a bag-of-words model and a vocabulary tree, and finally performing loop closure detection by judging the similarity between two frames of images in the bag-of-words model.

[0007] The technical solution provided by this invention is as follows:

[0008] An improved visual SLAM loop closure detection method based on image patch region feature points includes the following steps:

[0009] Step 1: Obtain the image data of the current frame, convert the image data of the current frame into a grayscale image, and divide the grayscale image of the current frame into different sub-image blocks.

[0010] Step 2: Divide the entire image into a grid according to the size of the input image in the current frame. Preferably, to limit the size of image blocks and ensure that they reflect the gradient direction histogram information of feature points in the image block, the grid size is adaptively divided based on the size of the input image. That is, for an image of size h×w, a grid of h / 10×w / 10 is used to divide it.

[0011] Step 3: Extract feature points from the divided grayscale image blocks and calculate the gradient direction histogram, setting its gradient magnitude to p(x, y) and gradient direction to θ(x, y). Therefore, the gradient vector of the feature points is as follows:

[0012]

[0013]

[0014] Step 4: Obtain h / 10×w / 10 image blocks by grid division, calculate the gradient direction histogram of feature points in each image block, and count the gradient vector of feature points in each image block.

[0015] Step 5: After calculating the gradient vectors of feature points in the image patch, a weight set W is used to represent the total weight of the feature points in the image patch, and the gradient direction histogram vector set W = {w1, w2, ..., w...} is set. n}, with mean weight w m .

[0016] Step 6: Calculate the weights of the gradient vectors of feature points in any image patch W in the current frame and its adjacent (upper, lower, left, and right) image patches. If the gradient vector weights W reach the mean weights w... m If it is valid, it is considered a valid image block; otherwise, it is considered an invalid image block.

[0017] Step 7: Starting with valid image patches, search for those that satisfy the mean weight w. m Other image patches are added to the weight set, and for each newly added valid weight value, the mean weight w is updated. m This continues until no new valid image blocks can be added.

[0018] Step 8: Set the corresponding image sequence and image block set number contained in the current frame, image block and feature points, and then obtain the feature vector of the effective image block by normalizing the gradient direction histogram vector of the effective image block in the current frame into a unit vector.

[0019] Step 9: Add the valid image patch information of the current frame to the bag-of-words model to build a vocabulary.

[0020] Step 10: When the next frame arrives, repeat the above steps, and then determine whether the current valid image block has a similarity to the image frames in the bag-of-words model that reaches the weight threshold. If the condition is met, it is considered a valid loop closure detection.

[0021] Preferably, in step seven, starting from the valid image blocks, a search is conducted to find those that satisfy the mean weight w. m Other image patches, the mean weight w m The weights are dynamic; the mean weight w is dynamically updated when other valid image patches are added. m .

[0022] Preferably, in step eight, the feature vector of the effective image block includes the size and orientation of the feature, which serves as a descriptor of the region feature and is an important factor in loop closure detection.

[0023] Preferably, in step ten, the weight threshold is a set weight parameter. When two frames of images are greater than or equal to the set weight parameter, they are considered as valid loop closure detection.

[0024] The beneficial effects of this invention are:

[0025] (1) This invention uses feature points of effective image blocks for extraction and description, which can avoid the algorithm wasting a lot of time on invalid image blocks with inconspicuous features, and greatly improve the real-time efficiency of the algorithm.

[0026] (2) This invention uses the bag-of-words model to measure the similarity between images. Through the iteration of the algorithm, the vocabulary tree in the bag-of-words model is continuously updated, so that the algorithm can accurately and efficiently achieve loop closure detection.

[0027] (3) In addition, the present invention can also be applied to loop detection in fields such as disinfection and epidemic prevention robots, warehousing and logistics robots, unmanned autonomous driving, AR / VR, and military rescue. Attached Figure Description

[0028] To make the technical solutions in the embodiments of the present invention clearer, the accompanying drawings required in the description of the embodiments will be introduced in a simple and detailed manner below.

[0029] Figure 1 This is a block diagram of an improved visual SLAM loop closure detection method based on feature points in image patch regions;

[0030] Figure 2 A schematic diagram illustrating the grid division of the input image;

[0031] Figure 3 This is a schematic diagram of the gradient position and direction vectors of feature points in an image patch.

[0032] Figure 4 This is a diagram illustrating valid and invalid image blocks;

[0033] Figure 5 The weight values ​​for the effective image blocks in each frame;

[0034] Figure 6 This is a schematic diagram of the vocabulary tree structure in the bag-of-words model. Detailed Implementation

[0035] To enable a better understanding of the method of the present invention, the technical solutions of the present invention will be described in simple and detailed form below with reference to the accompanying drawings in the embodiments of the present invention.

[0036] Example 1

[0037] Please see Figure 1As shown, this invention provides an improved visual SLAM loop closure detection method based on feature points in image patch regions, comprising the following steps:

[0038] Preparation: such as Figure 2 As shown in the diagram, when the camera captures a frame of image data, the image information data is in RGB format. The RGB image is then converted to a grayscale image. Each pixel in a grayscale image has a corresponding numerical value, ranging from 0 to 255. 0 represents black, and 255 represents white. The pixel value at a specific location can be obtained by locating the horizontal and vertical coordinates of the pixel grid.

[0039] Please see Figure 2 As shown, step one is to acquire the image data of the current frame, convert the image data of the current frame into a grayscale image, and divide the grayscale image of the current frame into different sub-image blocks.

[0040] Step 2: Divide the entire image into a grid based on the size of the input image in the current frame. To limit the size of image blocks and ensure that they reflect the gradient direction histogram information of feature points in the image blocks, the grid size is adaptively divided according to the size of the input image. That is, for an image of size h×w, a grid of h / 10×w / 10 is used to divide it.

[0041] Specifically, the input image size is 600×800, and it is divided using a 60×80 grid.

[0042] Please see Figure 3 As shown in the diagram, step three involves extracting feature points from the divided grayscale image blocks and calculating the gradient direction histogram, setting its gradient magnitude to p(x, y) and gradient direction to θ(x, y). Therefore, the gradient vectors of the feature points are as follows:

[0043]

[0044]

[0045] Specifically, the green area represents the divided image patch. The gradient direction histogram is calculated for the feature points in the image patch, including calculating their position, size, and direction, thereby obtaining the gradient vector of the feature points.

[0046] Step 4: Obtain h / 10×w / 10 image blocks by grid division, calculate the gradient direction histogram of feature points in each image block, and count the gradient vector of feature points in each image block.

[0047] Please see Figure 4As shown in the diagram, in step five, after calculating the gradient vectors of feature points in the image patch, a weight set W is used to represent the total weight of the feature points in the image patch, and the gradient direction histogram vector set W = {w1, w2, ..., w...} is set. n}, with mean weight w m .

[0048] Step 6: Calculate the weights of the gradient vectors of feature points in any image patch W in the current frame and its adjacent (upper, lower, left, and right) image patches. If the gradient vector weights W reach the mean weights w... m If it is valid, it is considered a valid image block; otherwise, it is considered an invalid image block.

[0049] Specifically, the green area is considered a valid image patch because the weight W of the green area reaches the mean weight w. m The blue area, which does not reach the mean weight, is considered an invalid image block.

[0050] Please see Figure 5 As shown in the diagram, step seven involves searching for valid image patches that satisfy the mean weight w. m Other image patches are added to the weight set, and for each newly added valid weight value, the mean weight w is updated. m This continues until no new valid image blocks can be added.

[0051] Step 8: Set the corresponding image sequence and image block set number contained in the current frame, image block and feature points, and then obtain the feature vector of the effective image block by normalizing the gradient direction histogram vector of the effective image block in the current frame into a unit vector.

[0052] Please see Figure 6 As shown in the diagram, step nine involves adding the valid image block information of the current frame into the bag-of-words model to construct a vocabulary tree.

[0053] Specifically, the image frames acquired by the camera continuously build and update the node information of the vocabulary tree, and the node information of the vocabulary tree is used to construct and update the features of the effective image blocks.

[0054] Step 10: When the next frame arrives, repeat the above steps, and then determine whether the current valid image block has a similarity to the image frames in the bag-of-words model that reaches the weight threshold. If the condition is met, it is considered a valid loop closure detection.

[0055] Example 2

[0056] This embodiment, based on the inventive technical solution, utilizes a handheld Kinect RGB-D camera with a resolution of 640×480 to collect 1700 sets of data in an indoor environment for model training and validation. A comparative experiment with the traditional ORB algorithm (Oriented FAST and Rotated BRIEF) demonstrates the overall effectiveness and accuracy of the method by comparing the model training results with the actual loop closure similarity scores. Because current algorithms cannot accurately determine whether two images are similar or were taken from the same location or angle as the human brain, perceptual bias and perceptual variation can occur. Therefore, we choose to simultaneously compare recall and precision to evaluate the model's effectiveness, calculated as follows:

[0057] Precision = Number of correctly extracted loop closure frames / Total number of extracted loop closure frames

[0058] Recall = Number of correctly retrieved loopback frames / Number of loopback frames in the sample

[0059] Accuracy describes the probability that all loop closures extracted by the algorithm are indeed true loop closures. Recall, on the other hand, refers to the probability that all true loop closures are correctly detected. The comparison results are as follows:

[0060] Table 1 Algorithm Comparison

[0061] method Precision Recall Orb (Baseline) 0.683 0.772 PatchUp(ours) 0.718 0.816

[0062] Table 1 compares the proposed PatchUp method with the traditional Orb algorithm. Because the Orb algorithm uses binary descriptors, it is more sensitive to indoor image noise and changes in ambient brightness, resulting in lower accuracy in loop closure detection compared to the proposed PatchUp method for feature points in image patch regions. This method increases the probability of identifying true loop closures to 0.718, demonstrating its superior accuracy. Furthermore, the Orb algorithm is weaker in handling scale variations in indoor environmental images, and it is less efficient and stable than the proposed PatchUp method when extracting feature points from handheld RGB-D cameras with significant rotational changes. The recall rate of 0.816 further illustrates the superior efficiency of the proposed method.

Claims

1. An improved visual SLAM loop closure detection method based on feature points of image patch regions, characterized in that, Includes the following steps: Step 1: Obtain the image data of the current frame, convert the image data of the current frame into a grayscale image, and divide the grayscale image of the current frame into different sub-image blocks; Step 2: Divide the entire image into a grid based on the size of the input image in the current frame; Step 3: Extract feature points from the divided grayscale image blocks and calculate the gradient direction histogram, while setting its gradient magnitude to p(x,y) and gradient direction to θ(x,y). Step 4: Obtain h / 10×w / 10 image blocks through grid division, calculate the gradient direction histogram of feature points in each image block, and count the gradient vector of feature points in each image block; Step 5: After calculating the gradient vectors of feature points in the image patch, a weight set W is used to represent the total weight of the feature points in the image patch, and the gradient direction histogram vector set W = {w1, w2, ..., w n }, with mean weight w m ; Step 6: Calculate the weights of the gradient vectors of feature points in any image block Wi in the current frame and its adjacent upper, lower, left, and right image blocks. If the gradient vector weights Wi reach the mean weights w... m If it is true, it is considered a valid image block; otherwise, it is considered an invalid image block. Step 7: Starting with valid image patches, search for those that satisfy and exceed the mean weight w. m Other image patches are added to the weight set, and for each newly added valid weight value, the mean weight w is updated. m Continue until no new valid image blocks can be added; Step 8: Set the corresponding image sequence and image block set number contained in the current frame, image block and feature point, and then obtain the feature vector of the effective image block by normalizing the gradient direction histogram vector of the effective image block in the current frame into a unit vector. Step 9: Add the valid image patch information of the current frame to the bag-of-words model to construct the vocabulary tree; Step 10: When the next frame arrives, repeat the above steps, and then determine whether the current valid image block has a similarity to the image frames in the bag-of-words model that reaches the weight threshold. If the condition is met, it is considered a valid loop closure detection.

2. The improved visual SLAM loop closure detection method based on image patch region feature points as described in claim 1, characterized in that, In steps one and three, the grayscale image is converted from the RGB image. The image data acquired by the camera is an RGB image, which is then converted to grayscale to enhance local feature information.

3. The improved visual SLAM loop closure detection method based on image patch region feature points as described in claim 1, characterized in that, In step two, the grid size is adaptively divided based on the input image size. That is, for an image of size h×w, a grid of h / 10×w / 10 is used to divide it.

4. The improved visual SLAM loop closure detection method based on image patch region feature points as described in claim 1, characterized in that, In step three, the gradient vectors of the feature points are as follows:

5. The improved visual SLAM loop closure detection method based on image patch region feature points as described in claim 1, characterized in that, In step seven, starting with valid image patches, we search for those that satisfy the mean weight w. m Other image patches, the mean weight w m The weights are dynamic; the mean weight w is dynamically updated when other valid image patches are added. m .

6. The improved visual SLAM loop closure detection method based on image patch region feature points as described in claim 1, characterized in that, In step eight, the feature vector of the effective image patch includes the size and orientation of the feature, which serves as a descriptor of the region feature and is an important factor in loop closure detection.

7. The improved visual SLAM loop closure detection method based on image patch region feature points as described in claim 1, characterized in that, In step nine, the bag-of-words model is composed of each valid image block extracted from the current frame image and its corresponding feature vector, and the vocabulary tree is composed of each corresponding frame image sequence and image block set number.

8. The improved visual SLAM loop closure detection method based on image patch region feature points as described in claim 7, characterized in that, In step nine, the node information of the vocabulary tree is continuously constructed and updated based on the acquired image frames. The node information of the vocabulary tree is used to construct and update the features of valid image blocks.

9. The improved visual SLAM loop closure detection method based on image patch region feature points as described in claim 1, characterized in that, In step ten, the weight threshold is a set weight parameter. When two frames of images are greater than or equal to the set weight parameter, they are considered to be valid loop closure detections.

Images