SLAM loopback detection method based on deep learning
By using EfficientNetV2 convolutional neural network in visual SLAM for global feature extraction and bag-of-word model similarity calculation, the problem of poor results and large calculations in the prior art in lighting changes and complex scenarios is solved, and more efficient and real-time loop detection is achieved.
Patent Information
- Application Number
- CN202411947660.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-03-28
AI Technical Summary
The existing visual SLAM loopback detection method is poor in lighting changes and complex scenarios, with large calculations and affecting real-time performance.
The SLAM loopback detection method based on deep learning is adopted, and the dictionary tree database is constructed using EfficientNetV2 convolutional neural network, and the similarity is calculated through global feature extraction and bag-of-word model for loopback detection.
It improves the accuracy and real-time performance of loopback detection, and can reliably extract global features in lighting changes and complex scenarios, reducing the computational complexity.
Smart Images
Figure CN119719399A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of visual monitoring methods, and in particular to a SLAM loop detection method based on deep learning. Background Art
[0002] With the continued development of unmanned systems, visual simultaneous localization and mapping (SLAM) has received significant attention. Visual SLAM uses image information from different, specialized cameras to achieve functions such as establishing the environment and estimating the system's own motion during motion, even without prior knowledge of the surrounding environment. Ensuring and improving the accuracy of unmanned systems' positioning and mapping in complex environments is a key application foundation for the industrialization of visual SLAM. Unmanned systems use the robot's own position and visual sensors to estimate motion in an unknown environment, intelligently localizing themselves and mapping the surrounding environment. This process is known as SLAM, or simultaneous localization and mapping.
[0003] The most commonly used loop detection method in visual SLAM currently uses a local feature-based dictionary tree to query related visual words. Traditional loop detection methods primarily rely on bag-of-words (BoW) algorithms. While relatively simple, these methods require extensive manual feature set to refine the algorithm, significantly limiting the localization and image composition performance of unmanned systems and causing performance discrepancies when significant lighting changes occur. Furthermore, these methods rely solely on local image information to extract and describe feature points. In some tasks, global information (such as the overall scene layout and context) can help improve matching accuracy. However, traditional methods fail to leverage this global information, making feature matching errors more likely when dealing with scenes with large variations. Furthermore, loop detection requires similarity calculations between the current frame and all previous keyframes. This computational complexity can rapidly increase in large-scale scenes, impacting the system's real-time performance. Summary of the Invention
[0004] In order to overcome the above problems existing in the prior art, the present invention proposes a SLAM loop detection method based on deep learning.
[0005] The present invention solves the technical problem by adopting a technical solution: a SLAM loop detection method based on deep learning, comprising the following steps: Step 1: Build a dictionary tree database based on convolutional neural network; Step 2: Input the image to be detected into the convolutional neural network to obtain the global features of the current image; Step 3: Based on the global features of the current image obtained in step 2, the similarity between the current image and the historical images is calculated using the dictionary database obtained in step 1; Step 4: If the similarity calculated in step 3 is higher than the set threshold, it is considered that a loop is detected, and the map is updated according to the loop detection result.
[0006] In the above-mentioned SLAM loop detection method based on deep learning, step 1 is specifically as follows:
[0007] Step 1.1: Use the EfficientNetV2 convolutional neural network to process the existing dataset or the historical image dataset to obtain the global feature vectors of all images in the dataset, and cluster the global feature vectors of all images to generate K cluster centers V1, V2, ..., V k , each cluster center represents a global visual word: ; Among them, C k is the kth cluster, N k is the number of global eigenvectors belonging to the cluster, F(I i ) is the global feature vector of the i-th image;
[0008] Step 1.2: The global feature vector F(I i ) is matched with the global visual word to generate a dictionary vector , where h k is image I i The degree of match with the global visual word is calculated based on the number of matches: ; in, is the indicator function, when F(I i ) is allocated in cluster V k When, h k The value is increased by one, j is the coordinate of the output feature map, and n is the coordinate of the convolution kernel.
[0009] In the above-mentioned SLAM loop detection method based on deep learning, the convolutional neural network is an EfficientNetV2 convolutional neural network.
[0010] In the above-mentioned deep learning-based SLAM loop detection method, the specific process of the EfficientNetV2 convolutional neural network acquiring global features is as follows: Step a: preprocess the image, including scaling, normalization, and data enhancement; Step b: The image obtained in step a is passed through multiple convolutional layers to extract local features, and high-level feature representations are gradually constructed by stacking depth-wise convolutional blocks; In step c, the global average pooling layer is used to construct the global features of the image, and finally a vector representing the global features of the image is obtained.
[0011] The above-mentioned SLAM loop detection method based on deep learning, the step b stacking deep convolution blocks includes deep convolution, point-by-point convolution, and SE module, the deep convolution performs convolution on each channel independently; the point-by-point convolution is used to transform the channel dimension; the SE module captures the correlation between channels through global pooling.
[0012] In the above-mentioned SLAM loop detection method based on deep learning, the similarity calculation formula in step 3 is specifically: ; Among them, h(I i ) is an image The bag-of-words vector, h(I j ) is an image Bag-of-words vector, Sim(I i ,I j ) is the similarity value, ‖h(I i )‖ is the image The bag-of-words vector modulus, ‖h(I j )‖ is the image The bag-of-words vector norm.
[0013] The beneficial effect of the present invention is that it optimizes loop detection technology based on EfficientNetV2. Compared with traditional convolutional neural networks, EfficientNetV2 uses new activation functions and regularization strategies to improve the generalization ability and stability of the model, so that it can still reliably extract discriminative global features under conditions such as lighting changes and different scene complexities. In addition, the introduction of improved high-efficiency modules ensures that the system significantly reduces computational complexity while maintaining accuracy, effectively improving the real-time performance of loop detection. Finally, EfficientNetV2 optimizes the depth, width, and resolution of the network through a scaling strategy. It can extract global features of the image with fewer parameters and computational complexity while maintaining high accuracy. This efficient feature extraction capability enables it to capture richer global contextual information and improve the accuracy of loop detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 It is a schematic diagram of the present invention. DETAILED DESCRIPTION
[0015] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0016] The present invention discloses a SLAM loop detection method based on deep learning, which mainly uses deep learning to optimize the feature extraction and loop detection algorithm in the system. The SLAM framework includes an offline vocabulary tree acquisition channel and an online visual SLAM operation channel, such as Figure 1 As shown in the figure, the light blue background is the visual SLAM operation channel, and the light green background is the dictionary tree acquisition channel. Unlike traditional visual SLAM methods, this method uses the EfficientNetV2 convolutional neural network architecture in deep learning to obtain global image features and construct a dictionary tree database using these features. This improves the loop detection method and enhances the accuracy and efficiency of unmanned system positioning and mapping.
[0017] The SLAM loop detection method includes the following steps:
[0018] Step 1: Build a dictionary tree database based on convolutional neural network.
[0019] Step 1.1: Use the EfficientNetV2 convolutional neural network to process the existing dataset or the historical image dataset to obtain the global feature vectors of all images in the dataset, and cluster the global feature vectors of all images to generate K cluster centers V1, V2, ..., V k , each cluster center represents a global visual word: ; Among them, C k is the kth cluster, N k is the number of global eigenvectors belonging to the cluster, F(I i ) is the global feature vector of the i-th image; Step 1.2: The global feature vector F(I i ) is matched with the global visual word to generate a dictionary vector , where h k is image I i The degree of match with the global visual word is calculated based on the number of matches: ; in, is the indicator function, when F(I i ) is allocated in cluster V k When, h k The value is increased by one, j is the coordinate of the output feature map, and n is the coordinate of the convolution kernel.
[0020] Step 2: Input the image to be detected into the convolutional neural network to obtain the global features of the current image; Step 3: Based on the global features of the current image obtained in step 2, the similarity between the current image and the historical images is calculated using the dictionary database obtained in step 1; In the loop detection module, the bag-of-words model is used to calculate the current image I i Compared with the previous frame image I j The process uses cosine similarity to calculate the similarity between the bag-of-words vectors of two images: ; Among them, h(I i ) is an image The bag-of-words vector, h(I j ) is an image Bag-of-words vector, Sim(I i ,I j ) is the similarity value, ‖h(I i )‖ is the image The bag-of-words vector modulus, ‖h(I j )‖image The bag-of-words vector norm.
[0021] Step 4: If the similarity calculated in step 3 is higher than the set threshold, it is considered that a loop is detected, and the map is updated according to the loop detection result.
[0022] The specific process of extracting global feature vectors using the EfficientNetV2 convolutional neural network includes: 1. Preprocessing images (1) Image scaling: Scale the input image I to the input size required by EfficientNetV2, such as 224×224: .
[0023] (2) Normalization: Normalize the image pixel values to the range required by the network to match the preprocessing method of the network during training. Assuming the image pixel value range is [0, 255], the normalization formula is: ; in, and are the mean and standard deviation used for normalization, respectively.
[0024] (3) Data enhancement: cropping, rotating, flipping, etc. of images to improve the generalization ability of the model.
[0025] 2. Extract features through EfficientNetV2 EfficientNetV2 extracts local features through multiple convolutional layers and gradually builds high-level feature representations by stacking depth-wise convolutional blocks.
[0026] (1) Convolution operation
[0027] The convolution layer is the basis of feature extraction. Assuming the input feature map is X and the convolution kernel is W, the formula for the convolution operation is: ; Among them, b is the bias term, i, j represent the coordinates of the output feature map, and m, n represent the coordinates of the convolution kernel.
[0028] (2) Improving image feature representation capabilities based on the MBConv module The MBConv (Mobile Inverted Bottleneck Conv) module is a key module of EfficientNetV2, which includes depthwise convolution, point-by-point convolution, SE module (Squeeze and Excitation) and other links.
[0029] (1) Depth convolution: Perform depth convolution on the input feature map X and the convolution kernel W: .
[0030] Depthwise convolution operates by performing convolution on each channel independently.
[0031] (2) Point-by-point convolution: Point-by-point convolution is a 1×1 convolution used for channel dimension transformation: ; Where c is the channel index.
[0032] (3) SE module: This module captures the correlation between channels through global pooling.
[0033] Global average pooling stage: .
[0034] Channel weight calculation step: ; in and is the weight matrix of the fully connected layer, is the Sigmoid activation function.
[0035] Channel weighting link: The calculated weight s is used to weight each channel: .
[0036] (3) Global pooling
[0037] At the end of EfficientNetV2, a global average pooling layer is used to construct the global features of the image. Global pooling compresses the feature map into a vector of fixed size.
[0038] Global average pooling: For the input feature map , the global average pooling is: ; in, and are the height and width of the feature map, respectively, and c is the channel index. This process de-averages the pixels of each channel, thereby compressing the spatial information into a global feature vector.
[0039] 3. Output feature vector
[0040] Through the above convolutional layer and pooling layer, we finally get a vector representing the global features of the image. Assuming that the dimension of the feature vector after global pooling is F, the global feature vector can be expressed as: ; Among them, C is the number of channels of the last layer feature map, each G C It is the feature value obtained by global average pooling. This feature vector can represent the global features of the image and is used for subsequent loop detection.
[0041] This paper combines the traditional ORB-SLAM3 scheme with deep learning and proposes a visual word bag model construction method based on global features. It focuses on the automated neural network architecture search technology and multi-scale feature fusion mechanism of the EfficientNetV2 algorithm to promote the comprehensiveness and accuracy of image feature extraction.
[0042] Based on the composite scaling strategy of the EfficientNetV2 algorithm, the network depth, width, and resolution are optimized, fully utilizing the advantages of lightweight design to efficiently extract features while reducing computational overhead, improving the real-time performance of the system, and perfecting the loop detection link to achieve high-precision and efficient positioning and mapping.
[0043] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the spirit and scope of protection of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present invention.
Claims
1. A SLAM loop detection method based on deep learning, characterized in that: The steps include: Step 1, construct a dictionary tree database based on convolutional neural network; Step 2: Input the image to be detected into the convolutional neural network to obtain the global features of the current image; Step 3, based on the global features of the current image obtained in step 2, the similarity between the current image and the historical image is calculated using the dictionary database obtained in step 1; Step 4: If the similarity calculated in step 3 is higher than the set threshold, it is considered that a loop is detected, and the map is updated according to the loop detection result.
2. A SLAM loop detection method based on deep learning according to claim 1, characterized in that, The step 1 is specifically as follows: Step 1.1, use the EfficientNetV2 convolutional neural network to process the existing data set or the historical image data set, obtain the global feature vectors of all images in the data set, and cluster the global feature vectors of all images to generate K cluster centers V1, V2, ..., V k , each cluster center represents a global visual word: ; Among them, C k is the kth cluster, N k is the number of global eigenvectors belonging to the cluster, F(I i ) is the global feature vector of the i-th image; Step 1.2: The global feature vector F(I i ) is matched with the global visual word to generate a dictionary vector , where h k is image I i The degree of match with the global visual word is calculated based on the number of matches: ; in, is the indicator function, when F(I i ) is allocated in cluster V k When k The value is increased by one, j is the coordinate of the output feature map, and n is the coordinate of the convolution kernel.
3. A SLAM loop detection method based on deep learning according to claim 1, characterized in that, The convolutional neural network is an EfficientNetV2 convolutional neural network.
4. A SLAM loop detection method based on deep learning according to claim 3, characterized in that, The specific process of obtaining global features by the EfficientNetV2 convolutional neural network is as follows: Step a, preprocessing the image, including scaling, normalization, and data enhancement; Step b: The image obtained in step a is subjected to multiple convolutional layers to extract local features, and a high-level feature representation is gradually constructed by stacking deep convolutional blocks; In step c, the global average pooling layer is used to construct the global features of the image, and finally a vector representing the global features of the image is obtained.
5. A SLAM loop detection method based on deep learning according to claim 4, characterized in that, The stacked deep convolution blocks in step b include deep convolution, point-by-point convolution, and SE modules. The deep convolution performs convolution on each channel independently; the point-by-point convolution is used to transform the channel dimension; and the SE module captures the correlation between channels through global pooling.
6. A SLAM loop detection method based on deep learning according to claim 1, characterized in that: The similarity calculation formula in step 3 is specifically: ; Among them, h(I i ) is the image The bag-of-words vector, h(I j ) is the image The bag-of-words vector, Sim(I i ,I j ) is the similarity value, ‖h(I i )‖ is the image The bag-of-words vector norm, ‖h(I j )‖image The bag-of-words vector norm.