Visual place identification and loopback detection method based on 2D-3D mixed feature enhancement

Abstract

The invention discloses a visual location identification and loopback detection method based on 2D-3D mixed feature enhancement. The method comprises the following steps: constructing a 2D-3D hybrid recognition model comprising a 2D reference recognition network and a descriptor correction network based on hybrid 3D geometric structure features, and training to perform 2D-3D feature fusion; and inputting the 2D image of the current place into the trained model, identifying the position of the current place according to the extracted to-be-identified 3D geometric feature enhanced global feature descriptor, and performing loopback detection in real time. When facing challenging scenes such as environment and view angle change, the method of the invention significantly improves the robustness and accuracy of visual place identification, and provides a solution with higher universality and high efficiency for loopback detection; according to the method, potential complementary characteristics between 2D semantic features and 3D geometric structure features are mined, a universal visual location recognition architecture with 2D-3D feature fusion is provided, and the performance of the 2D visual location recognition and loopback detection method can be improved.

Description

Technical Field

[0001] The present invention relates to a visual place recognition method, which relates to the technical fields of computer vision and back-end processing technologies such as loop detection in simultaneous localization and mapping of robots. Specifically, it relates to a visual place recognition and loop detection method based on enhanced 2D-3D hybrid features. Background Art

[0002] Simultaneous Localization And Mapping (SLAM) has always been one of the core capabilities for intelligent robot systems to achieve high automation. However, there are inevitable cumulative errors in the visual odometry at the front end of current SLAM, which often lead to trajectory drift during the long-term SLAM process of the robot, causing the robot to have an incorrect estimate of its own state, and further seriously affecting the operation performance of the robot and even causing catastrophic consequences. As one of the key capabilities in visual SLAM, visual place recognition can complete loop detection by comparing the visual similarity between visual perception data at the current position and historical data, thereby establishing long-term perceptual data association, eliminating cumulative errors during the long-term SLAM process, and further improving the autonomy, reliability, and stability of autonomous mobile robots in long-term operation tasks.

[0003] In recent years, with the continuous development of the fields of computer vision and robotics, visual place recognition technology has gradually received more and more attention and research. Currently, mainstream visual place recognition technologies extract local or global visual descriptors from environmental images, and determine whether there is a topological association (loop) between places by comparing the current descriptor with the descriptors in the historical database, and determine the approximate geographical location of the current. However, this technology faces many challenges in real loop scenarios of actual deployment applications. The most critical problem is that environmental factors such as the viewpoint, perspective, and lighting conditions of environmental perception often change, which will have a greater impact on visual perception data, and may further cause perceptual deviation and perceptual aliasing. This also leads to insufficient robustness of existing visual place recognition algorithms in the face of environmental and perspective changes, and the performance is also very limited. How to construct a better visual place recognition strategy to improve the above problems has always been the research focus in this field. Summary of the Invention

[0004] In order to solve the problems in the background art, the present invention provides a loop detection method based on a global descriptor enhanced with 2D-3D hybrid features.

[0005] The technical solution adopted by the present invention is:

[0006] The visual place recognition and loop closure detection method based on 2D-3D hybrid feature enhancement of the present invention includes:

[0007] Step 1) Obtain 2D images of several different places, add the place coordinates as labels, and construct a training set.

[0008] Step 2) Construct a 2D-3D hybrid recognition model including a 2D baseline recognition network and a descriptor correction network based on hybrid 3D geometric structure features. Use the training set to train the 2D-3D hybrid recognition model for 2D-3D feature fusion, extract the original 2D visual feature map and 3D point cloud, process to obtain the globally feature descriptor enhanced by the original 3D geometric features, and input it into the loss function of the 2D-3D hybrid recognition model for processing until the loss function converges, and obtain the trained 2D-3D hybrid recognition model.

[0009] Step 3) Input the 2D image of the current place into the trained 2D-3D hybrid recognition model for processing, extract the globally feature descriptor enhanced by the 3D geometric features to be recognized, and use the visual place re-identification method to process and then recognize the position of the current place and perform loop closure detection in real time.

[0010] In the said step 1), take the place where one of the 2D images is located as the target place, construct the 2D images of the places whose distance from the target place does not exceed the preset distance threshold as positive samples, and construct the 2D images of the places whose distance from the target place exceeds the preset distance threshold as negative samples, so as to construct a training set; add the place coordinates near the center of the position to which each 2D image belongs as a label.

[0011] In the said step 2), the 2D baseline recognition network adopts a deep learning model including a vision network. The original 2D image is input into the 2D baseline recognition network for processing. First, after being processed by the vision network, the 2D visual feature map including color, appearance, semantics, etc. of the original 2D image is extracted. Finally, the 2D baseline recognition network aggregates the 2D visual feature map to obtain the 2D visual place recognition feature encoding of the original 2D image; convert the original 2D visual place recognition feature encoding into several original 3D point clouds and then input them into the descriptor correction network based on hybrid 3D geometric structure features for training.

[0012] In step 2), the descriptor correction network based on the hybrid 3D geometric structure features first voxelizes each original 3D point cloud P using a voxel feature extraction network to obtain a voxel feature pyramid, projects the voxel feature pyramid to the bird's eye view (BEV) perspective to obtain a bird's eye view feature map, and extracts several bird's eye view features of the 3D point cloud P from the bird's eye view feature map. At the same time, each original 3D point cloud P is used to obtain N spatially uniformly distributed sampled key points through the farthest point sampling (FPS) method. First, the 3D features of each sampled key point are extracted, and after being processed by a multilayer perceptron (MLP) together with the voxel feature pyramid and the bird's eye view features, N 3D geometric structure features are obtained. The fused and enhanced 2D-3D hybrid features contain both rich key point structure information and semantic information. At the same time, each sampled key point is projected onto the original 2D image to obtain the coordinates of each sampled key point in the original 2D image, and then bilinear interpolation is performed in the 2D visual feature map according to the coordinates of each sampled key point in the original 2D image to obtain N 2D semantic features. The N 2D semantic features and the N 3D geometric structure features are fused by methods such as the self-attention mechanism and then aggregated through a feature aggregation module such as NetVLAD to obtain the original 2D-3D hybrid structure feature encoding of each sampled key point, forming a corrected descriptor. Finally, the original 2D-3D hybrid structure feature encoding and the original 2D visual place recognition feature encoding are jointly fine-tuned, corrected, and fused using methods such as principal components analysis (PCA) to obtain a global feature descriptor enhanced by the original 3D geometric features. The voxel feature extraction network specifically uses a voxel set abstraction (VSA) module; the overall loss function specifically uses a multi-similarity loss function.

[0013] In step 3), the original 3D geometric feature enhanced global feature descriptors extracted from 2D images at different locations are used as the historical database. The 2D image of the current location is input into the trained 2D-3D hybrid recognition model for processing. After processing, the 3D geometric feature enhanced global feature descriptors to be recognized are extracted and matched with the historical database to determine whether there is a topological association between locations, determine whether the current location is a location that has been reached in the database before, identify the location to which the original 3D geometric feature enhanced global feature descriptor with the highest similarity and the closest distance belongs as the current location. If it continues to move after moving to the current location and judges moving to the current location again, it is judged that a loop has occurred in the current movement trajectory to achieve loop detection.

[0014] The method of the present invention extracts 3D-RGB point clouds from the input 2D image frames through the visual odometer at the SLAM front end. The 3D geometric features of the current environment are obtained through the point cloud feature extraction network, and a highly aggregated 2D-3D hybrid correction descriptor is constructed by combining 2D semantic features, effectively fine-tuning and correcting the original descriptors of the benchmark 2D visual location recognition module, so as to integrate the performance complementary advantages between 2D-3D features, improve the performance upper limit of the current 2D visual location recognition system and its performance and robustness in various challenging environments.

[0015] The visual location recognition and loop detection system applicable to the above method of the present invention includes:

[0016] A data acquisition module for acquiring 2D images at different locations.

[0017] A model construction module for constructing a 2D-3D hybrid recognition model including a 2D benchmark recognition network and a descriptor correction network based on a hybrid 3D geometric structure feature, and performing visual location recognition and loop detection processing on 2D images.

[0018] A recognition detection display module for displaying the visual location recognition and loop detection results obtained by the model construction module.

[0019] The electronic device of the present invention includes: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method as described above.

[0020] The computer-readable storage medium of the present invention stores program data thereon, and when the program data is executed by a processor, the method as described above is implemented.

[0021] The beneficial effects of the present invention are:

[0022] The present invention proposes a method that explores the performance gain of 3D features for traditional 2D visual place recognition algorithms, integrates the rich color, appearance, and semantic features in 2D images and the stable and robust geometric structure features of 3D point clouds, and successfully breaks through the performance limitations of traditional 2D visual place recognition models in scenarios such as environmental conditions and viewpoint changes. Compared with traditional 2D visual place recognition methods, the present invention has achieved a remarkable improvement in performance and robustness in various challenging practical application scenarios such as environmental and viewpoint changes, significantly enhancing the robustness and accuracy of visual place recognition. The present invention verifies the potential complementary characteristics between 2D visual place recognition features and 3D geometric structure features, and proposes a general visual place recognition architecture for 2D-3D feature fusion. This architecture can improve the performance of many traditional 2D visual place recognition and loop closure detection methods, and through ablation experiments, it is proved that this architecture can generally improve the performance of many traditional 2D visual place recognition methods, and provides a more general and efficient solution for loop closure detection; this method is expected to further promote the application of 3D perception information in subsequent visual place recognition research. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a framework diagram of the method of the present invention;

[0024] Figure 2 is a network architecture diagram in the specific implementation under the framework of the present invention;

[0025] Figure 3 is a comparison diagram of the accuracy-recall curves between the present invention and existing advanced methods;

[0026] Figure 4 is a result diagram of the accuracy-recall curve of the ablation experiment demonstrating the effectiveness of the method of the present invention;

[0027] Figure 5 is a schematic diagram of the test results of representative pictures selected in the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] To more clearly describe the purpose, technical solution, and advantages of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific cases. It should be understood that the following specific cases are only used to explain the present invention, and the scope of the present invention is not limited by the following cases. In addition, the technical features involved in each embodiment of the present invention can be combined with each other as long as they do not conflict with each other.

[0029] As Figure 1 and Figure 2 shown, the visual place recognition and loop closure detection method based on 2D-3D hybrid feature enhancement of the present invention is specifically as follows:

[0030] Step 1) Obtain 2D images at several different locations and add the location coordinates as labels. Take the location where one of the 2D images is located as the target location. Construct positive samples from each 2D image at locations whose distance from the target location does not exceed a preset distance threshold, and construct negative samples from each 2D image at locations whose distance from the target location exceeds the preset distance threshold, thus constructing a training set; add the location coordinates near the center of the location to which each 2D image belongs as a label.

[0031] Step 2) Construct a 2D-3D hybrid recognition model that includes a 2D baseline recognition network and a descriptor correction network based on a hybrid 3D geometric structure feature. The 2D baseline recognition network uses a deep learning model that includes a vision network. The original 2D image is input into the 2D baseline recognition network for processing. First, after being processed by the vision network, a 2D vision feature map of the original 2D image, including color, appearance, semantics, etc., is extracted. Finally, the 2D baseline recognition network aggregates the 2D vision feature map to obtain the 2D vision location recognition feature encoding of the original 2D image.

[0032] Specifically, the deep learning model uses the Cross-Image Correlation Aware Visual Place Recognition (CricaVPR) model. The vision network can specifically use deep neural networks such as the Visual Geometry Group (VGG), Residual Network (ResNet), self-supervised learning vision network DINO, and DINOv2 (self-DIstillation with NOlabels) to effectively extract high-quality visual features. In the 2D feature extraction stage, use the deep learning model to extract features in the 2D image. The deep network outputs a feature map in the form of a tensor, and effectively aggregates the features to output a feature encoding in the form of a vector, which is used as the original 2D vision location recognition feature encoding of the model.

[0033] The 2D baseline re-identification network is mainly responsible for extracting 2D visual feature maps such as color, appearance, and semantics of 2D images and aggregating them to generate an original descriptor. The quality of the original descriptor determines the baseline performance level of the entire fusion model. Therefore, the design of a high-performance 2D baseline re-identification network is crucial. In the specific implementation of the present invention, the 2D baseline re-identification network is based on the fine-tuned vision large model DINOv2, and combines multi-scale Generalized Mean Pooling (GeM) to complete the extraction and aggregation of 2D features, so as to achieve higher baseline visual place recognition performance. In the feature extraction stage, the DINOv2 model specialized for the visual place recognition task is used to extract the feature map of the image. The DINOv2 model is based on the Vision Transformer (ViT) to complete the extraction of visual features. Taking the self-supervised learning vision network DINOv2 as an example, the fine-tuned self-supervised learning vision network DINOv2 specialized for the visual place recognition task can be used to extract the feature map of the image. First, the pixel length and width of the input 2D RGB image are normalized to a tensor of H×W×C (224×224×3), and then divided into image patches of size P×P to form a sequence of M image patches. H×E / P 2 , and after each sequence is flattened, it is mapped through a linear projection into a vector of length D . A learnable classification token x cls is added before the image patch projection sequence, and combined with the learnable position embedding E pos to form the original input of the Vision Transformer (ViT) model. The input passes through the transformer encoder, which is alternately composed of a multi-head self-attention layer MHA (Multi-Head Attention), a multi-layer perceptron MLP, and a normalization layer LN (Layer Normalization). Based on the ViT model, the DINOv2 model has been pre-trained on a large amount of data using a self-supervised strategy, and can obtain rich general visual features in the image.

[0034] To further unleash the performance specialized for the visual place recognition task of the pre-trained general model, the model is fine-tuned using existing fine-tuning methods based on the original DINOv2 model to achieve performance optimization specialized for the visual place recognition task. For example, a lightweight multi-scale convolutional enhancement adaptation module can be used to enhance and improve the model's performance in the visual place recognition task. The addition of the adaptation module changes the structure of ViT, which mainly consists of an upper projection, multi-scale convolution, a lower projection, and skip connections. When training the model, the parameters of the ViT pre-trained model are kept frozen, and only the parameters in the lightweight adaptation module are trained to achieve performance optimization specialized for the visual place recognition task. After passing through the improved DINOv2 network, the original image will finally generate a feature map of dimension B×W×H×D. In the feature aggregation stage, as Figure 2 shown, the feature map of dimension B×W×H×D is subjected to multi-scale (1×1, 2×2, and 3×3) Generalized Mean pooling (GeM) feature aggregation, and finally a feature encoding of dimension 14×D is generated. Finally, feature enhancement of the feature encoding is performed through self-attention to serve as a descriptor of the original 2D semantic features.

[0035] After passing through the fine-tuned self-supervised learning visual network DINOv2, the original 2D image will finally generate a feature map of dimension B×W×H×D. The feature map undergoes efficient feature aggregation through feature aggregation modules such as Generalized Mean pooling (GeM) to generate a visual feature encoding of dimension M×D, and then feature enhancement is performed through the self-attention mechanism encoder, and finally the original 2D visual place recognition feature encoding is output

[0036] The original 2D visual place recognition feature encoding is transformed into a number of original 3D point clouds. As a pure vision-based place recognition system, the input end of the 3D module receives the point clouds generated by the front end of Simultaneous Localization and Mapping (SLAM), without the need to rely on additional sensors such as lidar. The process of generating the point clouds is as follows: the front end of the SLAM algorithm outputs the inverse depth map of the image, and the image is inverse-projected according to the inverse depth map to obtain the 3D point cloud of the environment, projecting the H×W pixel points in the 2D image into the 3D space, and generating the 3D point cloud P of the environment at the front end of SLAM * ∈R (H×W)×6 , where each point contains six-dimensional features, including geometric information XYZ and color information RGB. This method further restricts the range of the original point cloud on the X, Y, and Z axes, screens the point cloud with the number of H×W, and removes the projection results with excessive deviations such as the far end and the sky, and finally obtains a point cloud P∈R containing J points J×6As the input of the subsequent module. It should be noted that the generation of the entire point cloud can be regarded as one of the internal processes of the mapping process of the Simultaneous Localization and Mapping (SLAM) algorithm. This method uses the intermediate data of the SLAM process to complete the extraction of 3D structural features. Therefore, the data generation and preprocessing of the 3D module will not cause additional time and memory overhead.

[0037] Encode the original 2D visual place recognition features. Use the front-end algorithm of DROID-SLAM for Simultaneous Localization and Mapping, or 3D reconstruction techniques such as Structure from Motion (SfM), Visual Odometry, and mapping algorithms to obtain the (inverse) depth of the image pixels. According to the depth and the inverse projection model of the camera, inverse project the image pixels, thereby converting them into several initial 3D point clouds P'. Then, screen each initial 3D point cloud P', removing projection results with too large deviations such as those far away and the sky, that is, removing the point clouds whose distance from the origin is greater than the preset distance threshold. Finally, obtain the remaining several 3D point clouds P as the input of the subsequent module.

[0038] Then, input the original 3D point cloud into the descriptor correction network based on the hybrid 3D geometric structure features for training. Use the training set to train the 2D-3D hybrid recognition model for 2D-3D feature fusion, and extract the original 2D visual feature map and 3D point cloud. After processing, obtain the global feature descriptor enhanced by the original 3D geometric features and input it into the loss function of the 2D-3D hybrid recognition model for processing until the loss function converges, and obtain the trained 2D-3D hybrid recognition model.

[0039] The descriptor correction network based on the hybrid 3D geometric structure features first performs voxelization processing on each original 3D point cloud P using the voxel feature extraction network. In the specific implementation process, the input point cloud is first voxelized to generate L v ×W v ×H vFor the voxel grid, the voxel set abstraction module VSA is used to learn the voxel structure features, and a voxel feature pyramid is obtained; the voxel feature pyramid is projected onto the bird's-eye view (BEV) perspective to obtain a bird's-eye view feature map, and the bird's-eye view features of several 3D point clouds P are extracted from the bird's-eye view feature map; at the same time, each original 3D point cloud P is processed by the farthest point sampling (FPS) method to obtain N spatially uniformly distributed sampled key points. First, the 3D features of each sampled key point are extracted, and after being processed by a multi-layer perceptron (MLP) together with the voxel feature pyramid and the bird's-eye view features, N 3D geometric structure features are obtained. The fused and enhanced 2D-3D hybrid features contain both rich key point structure information and semantic information; at the same time, each sampled key point is projected onto the original 2D image to obtain the coordinates of each sampled key point in the original 2D image, and then after bilinear interpolation processing in the 2D visual feature map according to the coordinates of each sampled key point in the original 2D image, N 2D semantic features are obtained. The N 2D semantic features and the N 3D geometric structure features are fused by methods such as the self-attention mechanism and then aggregated by a feature aggregation module such as NetVLAD to obtain the original 2D-3D hybrid structure feature encoding of each sampled key point, forming a corrected descriptor. Finally, the original 2D-3D hybrid structure feature encoding and the original 2D visual place recognition feature encoding are jointly fine-tuned, corrected, and fused by methods such as principal component analysis (PCA) to obtain a global feature descriptor enhanced with the original 3D geometric features. The voxel feature extraction network specifically uses the voxel set abstraction module VSA; the overall loss function specifically uses the multi-similarity loss function.

[0040] As a pure vision-based place recognition system, the input end of the 3D module in the method of the present invention receives the point cloud generated by the three-dimensional reconstruction method without relying on additional sensors such as lidar. The descriptor correction module extracts stable environmental geometric structure features using the 3D point cloud generated by dense three-dimensional reconstruction. In order to more effectively integrate the complementary features from the 2D image and the 3D point cloud and further improve the quality of the structure features, the present invention designs a descriptor correction module based on the hybrid 3D geometric structure features. This module uses the 3D point cloud generated by the visual SLAM front end and the original 2D semantic features as inputs, enhances the 3D geometric features with the 2D semantic information, performs 2D-3D information fusion at the feature level to form hybrid structure features, and highly aggregates them to generate a corrected feature encoding based on the 2D-3D hybrid features. The corrected feature encoding can fine-tune the original descriptor of the 2D part, thereby playing a supplementary and corrective role without reducing the performance of the original 2D benchmark network and stably improving the performance.

[0041] Different from the simple splicing and fusion strategy at the descriptor level, the present invention realizes the fusion of 2D-3D information at the feature level. Specifically, the present invention further enhances the geometric structure features from the 3D point cloud using semantic and other visual information to generate a higher-quality 2D-3D hybrid feature encoding for correcting the 2D original descriptor. First, the input 3D point cloud is voxelized to generate a voxel grid of L v ×W v ×H v . A voxel convolutional neural network CNN (Convolutional Neural Network) with 3D sparse convolution is used for voxel structure feature learning, as shown in Figure 2 . The feature blocks are downsampled four times at a sampling rate of 2 times by means of sparse convolution to obtain feature pyramids at different scales. The finally 8-fold downsampled features are projected onto the bird's-eye view BEV perspective to obtain a bird's-eye feature map.

[0042] Furthermore, the voxel set abstraction module VSA of the present method efficiently extracts and aggregates the 3D structural features and 2D semantic features. First, 256 spatially uniformly distributed key points are obtained from the point cloud through the farthest point sampling FPS algorithm. For each key point, 3D structural features are extracted and projected into the 2D feature map generated by DINOv2 to extract 2D semantic features.

[0043] In terms of 3D structural features, the voxel set abstraction module VSA aggregates the 3D voxel features within a distance r k range near the key points. After obtaining the voxel feature set near the key points, the voxel features are further extracted by means of the PointNet module, that is, processed sequentially using a multi-layer perceptron MLP and a max pooling layer. Then the voxel set abstraction module VSA extracts multi-scale structural features f i pv from the 3D voxel CNN and combines them with the original point cloud features f i raw and the corresponding features f i bev in the bird's-eye BEV map to construct the corresponding key point structural features, and finally maps them through a multi-layer perceptron MLP to finally obtain a 620-dimensional point cloud structural feature.

[0044] The present method uses the semantic features of the key points to further enhance the structural features of the point cloud to form the final 3D features of the point cloud. In this process, first, the key points are projected back into the original image through the camera internal parameters to obtain the 2D coordinates of the key points, and then the corresponding 128-dimensional 2D semantic features fs of the key points are extracted from the feature map of the 2D module through bilinear interpolation and a fully connected layer. i, Finally, the 2D semantic features of the key points are concatenated with the 3D structural features and then weighted and fused through the self-attention mechanism to obtain the final 3D features of N×D dimensions. The 2D-3D hybrid features enhanced through the above process contain both rich key point structure information and semantic information. Feature fusion based on the attention mechanism enables the model to focus on areas such as buildings, road signs, and vegetation in the environment that contribute more significantly to the visual place recognition task, and reduce the attention to dynamic vehicles, pedestrians, and road surfaces that do not contribute or even cause negative interference to the visual place recognition task, thereby effectively improving the visual place recognition performance of the point cloud hybrid structure features. As Figure 3 shown, the 2D baseline skeleton uses DINOv2 as the feature extractor, uses the Generalized-Mean (GeM) and Feature Mixer methods as aggregators respectively, and uses the fusion method proposed by the present invention for performance enhancement to complete place recognition. Compare its effect with the self-supervised visual classifier (DINOv2-CLS), the deep learning-based local aggregation descriptor vector (Network for Vector of Locally Aggregated Descriptors, NetVLAD), (DINOv2 / VGG16 feature extractor), cross-image correlation-aware visual place recognition (Cross-image Correlation-aware Visual Place Recognition, CricaVPR), and feature mixing visual place recognition (Feature Mixing Visual Place Recognition, MixVPR), demonstrate the effect of this method compared with other advanced methods, and draw the accuracy-recall curves of various methods. It can be seen that this method has better performance in various indicators such as the area under the curve AUC (Area Under Curve) and the maximum recall rate R at an accuracy of 100% P100 and other indicators. It can be seen that the 2D-3D feature fusion strategy proposed by the present invention can more effectively integrate visual place recognition features of different modalities and achieve more effective performance improvement.

[0045] The fine-tuning and correction of the original 2D visual place recognition features are specifically as follows:

[0046] In order to use the extracted point cloud hybrid structure features to complete the environmental description, the features need to be further aggregated to form a corrected descriptor. Use a feature aggregation module NetVLAD network to highly aggregate the N×D-dimensional feature encoding to form a K×D-dimensional corrected descriptor (K << N). During the feature aggregation process, first generate K learnable clustering centers {c1,…,c K},c k ∈R D , for each key point i, its weight to the k-th clustering center is determined by soft assignment, and the finally aggregated hybrid feature corrected descriptor is obtained by the weighted sum of the assigned weight and the distance from the key point to the clustering center.

[0047] The generated corrected descriptor aims to fine-tune and correct the original descriptor of the 2D module by using the highly aggregated 3D point cloud hybrid structure features. Therefore, different from the 64 clustering centers commonly used in the feature aggregation module NetVLAD module in traditional 3D place recognition networks, there are only 4 clustering centers generated in this invention case, and the dimension of the finally generated highly aggregated corrected descriptor is 4×D to ensure that the hybrid structure features of the 3D module play the role of fine-tuning and correction.

[0048] During the descriptor correction process, different from the direct splicing strategy with equal weights, the hybrid features are encoded and highly aggregated into a corrected descriptor to ensure that this highly aggregated hybrid feature does not compress the performance of the original descriptor of the 2D module, but plays the role of fine-tuning, supplementing and correcting during the entire model training process. Using the above method, this study has achieved remarkable performance improvement on the basis of the 2D benchmark VPR network by fully integrating 3D information. The effect of this fine-tuning and correction is as Figure 3 shown. It can be seen that the descriptor with simple equal-weight splicing has the risk of dragging down the performance of the 2D benchmark in some cases, while the fine-tuning and correction strategy proposed in this invention fully integrates the effective information of 2D-3D place recognition and achieves remarkable performance improvement.

[0049] The visual place re-identification method is an image retrieval technology, that is, matching the currently observed image with the images observed at the places visited before, and thus identifying and locating the place; loop detection refers to a method in the simultaneous localization and mapping SLAM field of robots, which establishes data association for intelligent pose estimation by matching the currently observed image with the images observed at the places visited before to improve the accuracy of map construction and the stability of positioning; a descriptor is a representation method of 2D image features, usually abstracting the key features of an image into corresponding descriptors for place similarity judgment and matching in place re-identification technology.

[0050] The corrected descriptor is used to correct the original 2D semantic feature descriptor to obtain the global descriptor. The global feature encoding F g uses the original 2D semantic feature descriptor F of N×D dimensions 2D and the 2D-3D hybrid feature descriptor F of K×D dimensions 3DConnection structure. Methods such as principal component analysis (PCA) are used to complete feature fusion and dimensionality reduction, retain key features, and generate a global environment descriptor. By comparing the similarity between the descriptor of the current environment and the descriptors in the historical database, location recognition is completed to determine the approximate current geographical location and to judge the generation of loops in the agent's movement trajectory.

[0051] The process of generating a global descriptor for 2D-3D information fusion is as follows:

[0052] Principal component analysis (PCA) is used to compress the feature dimensions while maintaining the quality of the global descriptor. Global feature encoding F g Use the original 2D semantic feature descriptor F 2D ∈R 14×D and the 2D-3D hybrid feature descriptor F 3D ∈R K×D Connection structure. Use PCA for dimensionality reduction, retain key features, and generate a global environment descriptor f(F g ) ∈R G , which is used for subsequent visual location recognition and loop detection tasks.

[0053] Connect the two-dimensional and three-dimensional feature encodings and input them into the PCA module. Through the multi-similarity loss function for metric learning until the model converges, enabling the model to autonomously learn and balance the relationship between the original feature encoding and the refined feature encoding. The key two-dimensional visual location recognition features are retained, while the unimportant or redundant two-dimensional features are replaced by more stable three-dimensional structural features, thus achieving effective fusion of 2D-3D information and enhancing the performance and robustness of the descriptor in a perception-challenging environment.

[0054] Step 3) Use the global feature descriptors enhanced with the original 3D geometric features extracted from the 2D images of each different location as the historical database. Input the 2D image of the current location into the trained 2D-3D hybrid recognition model for processing. After processing, extract the global feature descriptor enhanced with the 3D geometric features to be recognized and match it with the historical database to judge whether there is a topological association between locations, determine whether the current location is a location that has been reached in the database, and identify the location to which the global feature descriptor enhanced with the original 3D geometric features with the highest similarity and the shortest distance belongs as the current location. If it continues to move after moving to the current location and judges moving to the current location again, it is judged that a loop has occurred in the current movement trajectory to achieve loop detection.

[0055] The method of the present invention extracts 3D-RGB point clouds from the input 2D image frames through the visual odometer of the SLAM front end. The 3D geometric features of the current environment are obtained through the point cloud feature extraction network, and the highly aggregated 2D-3D hybrid correction descriptor is constructed by combining the 2D semantic features, so as to effectively fine-tune and correct the original descriptors of the reference 2D visual place recognition module, thereby integrating the performance complementary advantages between the 2D-3D features, improving the performance upper limit of the current 2D visual place recognition system and the performance and robustness in various challenging environments.

[0056] The present invention also constructs a visual place recognition and loop closure detection system, including a data acquisition module, a model construction module, and an identification detection display module. The data acquisition module is used to acquire 2D images of different locations; the model construction module is used to construct a 2D-3D hybrid recognition model including a 2D reference recognition network and a descriptor correction network based on the hybrid 3D geometric structure features, and perform visual place recognition and loop closure detection processing on the 2D images; the identification detection display module displays the visual place recognition and loop closure detection results obtained by the model construction module.

[0057] As Figure 4 shown, it is an ablation experiment to prove the effectiveness of the method of the present invention. It can be seen that the method of the present invention can effectively exploit the complementary characteristics of 2D-3D information, and the present invention has obvious performance improvement compared with the pure 2D method, pure 3D method, and the scheme of simple splicing of 2D-3D. As Figure 5 shown, it is a schematic diagram of the test results of representative pictures selected in the embodiment of the present invention. In challenging scenarios such as changes in lighting conditions, weather changes, teaching viewpoints, occlusion, and extreme situations, the method of the present invention, the self-supervised visual classifier (DINOv-CLS), the local aggregation descriptor vector NetVLAD based on deep learning (VGG16 / DINOv2 feature extractor), the cross-image correlation-aware visual place recognition CricaVPR, and the feature-mixed visual place recognition MixVPR are used to identify the query pictures. The dotted box and the solid box are the successful recognition results and the failed recognition results respectively. It can be seen that the present invention has better performance than the existing methods in challenging scenarios such as changes in environmental conditions and viewpoint changes.

[0058] The above content only expresses the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. According to the technical idea proposed by the present invention, it should be understood that the protection scope of the present invention is not limited by the above content. The protection scope required by the present invention is defined by the appended claims and their equivalent systems, and any changes and improvements made on the basis of the technical solution belong to the protection scope of the claims of the present invention.

Claims

1. A visual place recognition and loop detection method based on 2D-3D hybrid feature enhancement, characterized in that: include: Step 1) Obtain 2D images of several different locations and add the location coordinates as labels to construct a training set; Step 2) constructing a 2D-3D hybrid recognition model including a 2D reference recognition network and a descriptor correction network based on hybrid 3D geometric structure features, using a training set to train the 2D-3D hybrid recognition model for 2D-3D feature fusion, and extracting the original 2D visual feature map and 3D point cloud, and after processing, obtaining the original 3D geometric feature enhanced global feature descriptor, inputting it into the loss function of the 2D-3D hybrid recognition model for processing until the loss function converges, and obtaining a trained 2D-3D hybrid recognition model; Step 3) The 2D image of the current location is input into the trained 2D-3D hybrid recognition model for processing, the global feature descriptor enhanced with the 3D geometric features to be recognized is extracted and processed using the visual position re-recognition method to identify the location of the current location and perform loop detection in real time.

2. The method for visual location recognition and loop detection based on 2D-3D hybrid feature enhancement according to claim 1, characterized in that: In the step 1), the location where one of the 2D images is located is taken as the target location, each 2D image of a location whose distance from the target location does not exceed a preset distance threshold is constructed as a positive sample, and each 2D image of a location whose distance from the target location exceeds the preset distance threshold is constructed as a negative sample, thereby constructing a training set; each 2D image is added with the location coordinates of the center of its location as a label.

3. The method for visual location recognition and loop detection based on 2D-3D hybrid feature enhancement according to claim 1, characterized in that: In the step 2), the 2D reference recognition network adopts a deep learning model including a visual network, and the original 2D image is input into the 2D reference recognition network for processing. First, the 2D visual feature map of the original 2D image is extracted after processing by the visual network, and finally the 2D reference recognition network aggregates the 2D visual feature map to obtain the 2D visual location recognition feature code of the original 2D image; the original 2D visual location recognition feature code is converted into several original 3D point clouds and then input into the descriptor correction network based on hybrid 3D geometric structure features for training.

4. The method for visual location recognition and loop detection based on 2D-3D hybrid feature enhancement according to claim 3, characterized in that: In the step 2), the descriptor correction network based on the hybrid 3D geometric structure features first voxelizes each original 3D point cloud P using a voxel feature extraction network to obtain a voxel feature pyramid, projects the voxel feature pyramid to a bird's-eye view BEV perspective to obtain a bird's-eye feature map, and extracts bird's-eye features of several 3D point clouds P from the bird's-eye feature map; at the same time, each original 3D point cloud P is subjected to a farthest sampling FPS method to obtain N spatially uniformly distributed sampling key points, first extracts the 3D features of each sampling key point, and processes them together with the voxel feature pyramid and the bird's-eye feature through a multi-layer perceptron MLP to obtain N 3D geometric structure features. feature; at the same time, each sampling key point is projected into the original 2D image to obtain the coordinates of each sampling key point in the original 2D image, and then bilinear interpolation processing is performed in the 2D visual feature map according to the coordinates of each sampling key point in the original 2D image to obtain N 2D semantic features, and the N 2D semantic features and N 3D geometric structure features are fused and then aggregated to obtain the original 2D-3D hybrid structure feature encoding of each sampling key point; finally, the original 2D-3D hybrid structure feature encoding is used to fine-tune and correct the original 2D visual place recognition feature encoding, and then the original 3D geometric feature enhanced global feature descriptor is obtained.

5. The method for visual location recognition and loop detection based on 2D-3D hybrid feature enhancement according to claim 1, characterized in that: In the step 3), the original 3D geometric feature enhanced global feature descriptors extracted from the 2D images of different locations are used as a historical database, and the 2D image of the current location is input into the trained 2D-3D hybrid recognition model for processing. After processing, the 3D geometric feature enhanced global feature descriptors to be identified are extracted and matched with the historical database, and the location to which the original 3D geometric feature enhanced global feature descriptors with the highest similarity and the closest distance belong is identified as the current location. If the location continues to move after moving to the current location and is judged to move to the current location again, it is judged that a loop has been generated in the current motion trajectory to achieve loop detection.

6. A visual location recognition and loop detection system suitable for the method according to any one of claims 1 to 5, characterized in that: include: A data acquisition module, used to acquire 2D images of various locations; A model building module is used to build a 2D-3D hybrid recognition model including a 2D reference recognition network and a descriptor correction network based on hybrid 3D geometric structure features, and perform visual location recognition and loop detection on 2D images; The recognition detection display module displays the visual location recognition and loop detection results obtained by the model building module.

7. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images