Register auxiliary aggregation visual position identification method based on DINOv2

Abstract

The invention discloses a register-assisted aggregation visual position identification method based on DINOv2, and the method comprises the following steps: 1, collecting streetscape images in a data set, forming a training sample and a test sample, and preprocessing the training sample and the test sample; step 2, constructing a neural network comprising a multi-scale feature fusion module according to a DINOv2 network; putting the preprocessed training sample into an initial model, obtaining a multi-scale feature through the neural network, and obtaining a local descriptor at the same time; 3, training the neural network, and optimizing parameters of the neural network to obtain a trained model; and 4, inputting a test image in the test sample into the trained model to obtain a verification image matched with the test sample, and outputting a recall rate. According to the method, the representation capability of features under different resolutions is enhanced, the robustness of global descriptors is improved, and the accuracy of position retrieval is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a register-assisted aggregated visual position recognition method based on DINOv2. Background Art

[0002] Visual Place Recognition (VPR) aims to estimate the location information of the current query image by retrieving the image that best matches the query location from a database of visual scene images containing geolocation tags. VPR has long been widely used in mobile robots and augmented reality, such as autonomous driving, image geolocation, and 3D reconstruction. Its main challenges include changes in conditions, such as lighting, weather and seasons, changes in viewpoint, perceptual aliasing, and appearance changes over time.

[0003] A recent work, the Sinkhorn algorithm for local aggregate descriptors, SALAD, more efficiently allocates local features to global descriptors by remodeling the feature-to-cluster allocation problem of the optimal transfer algorithm. In order to further improve the discriminative ability of the global descriptor, SALAD introduces a "trash can" mechanism to allow the aggregation process to discard uninformative features. However, by extracting and analyzing the heat map results of SALAD retrieval images, it is found that the "trash can" mechanism in SALAD easily discards some robust information such as buildings as useless information, but still retains some dynamic non-robust information such as cars. Such a "trash can" mechanism will reduce the feature representation ability of the model, thereby reducing the retrieval performance of the model. Summary of the invention

[0004] The present invention aims to provide a register-assisted aggregated visual location recognition method based on DINOv2, which enhances the representation capability of features at different resolutions, improves the robustness of global descriptors, and improves the accuracy of location retrieval.

[0005] In order to achieve the purpose of the present invention, the present invention provides a register-assisted aggregated visual position recognition method based on DINOv2, comprising the following steps:

[0006] Step 1: Collect street view images in the data set to form training samples and test samples, and pre-process the training samples and the test samples;

[0007] Step 2: construct a neural network including a multi-scale feature fusion module based on the DINOv2 network; put the preprocessed training samples into the initial model, and obtain multi-scale features through the neural network, and obtain local descriptors;

[0008] Step 3: for the local descriptors with multi-scale features, cluster the local descriptors using the optimal transmission algorithm and aggregate the global descriptors with the aid of registers, train the neural network, optimize the neural network parameters, and obtain a trained model;

[0009] Step 4: Input the test image in the test sample into the trained model to obtain a verification image matching the test sample, and output the recall rate.

[0010] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the register-assisted aggregated visual position recognition method based on DINOv2 is implemented.

[0011] A non-transitory computer-readable storage medium, characterized in that the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned register-assisted aggregated visual position recognition method based on DINOv2.

[0012] A computer program product includes computer program instructions. When the computer program instructions are executed on a computer, the computer is caused to execute the above-mentioned register-assisted aggregated visual position recognition method based on DINOv2.

[0013] Compared with the prior art, the significant progress of the present invention lies in: (1) The present invention proposes a multi-scale feature fusion module to enhance the representation capability of features at different resolutions by fusing the multi-scale features of the input image; (2) The present invention proposes a register-assisted aggregation module to filter out global background features and dynamic non-robust features in the image, thereby effectively improving the robustness of the global descriptor and improving the accuracy of location retrieval; (3) The experimental results of the present invention on five widely used datasets, namely MSLS, Pittsburgh250k, Pittsburgh30k, NordLand and SPED, show that RegVPR is superior to the SOTA method.

[0014] In order to more clearly illustrate the functional characteristics and structural parameters of the present invention, further description is given below in conjunction with the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0016] Figure 1 It is the overall architecture diagram of the present invention;

[0017] Figure 2 It is a comparison of the attention results between the present invention and the SALAD method. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0019] The present invention is a register-assisted aggregation visual position recognition method based on DINOv2, combined with Figure 1 , including the following steps:

[0020] Step 1: Collect street view images in the data set to form training samples and test samples, and pre-process the training samples and the test samples;

[0021] 1-1. Collect street view images from GSV-Cities, MSLS, Pittsburgh250k, Pittsburgh30k, NordLand, and SPED datasets, use the GSV-Cities dataset as training samples, and use the other datasets as test samples;

[0022] 1-2. Preprocessing the training samples and the test samples, wherein the preprocessing adjusts the resolution of the street view image to 224×224 by a bilinear interpolation method.

[0023] Step 2: construct a neural network including a multi-scale feature fusion module based on the DINOv2 network; put the preprocessed training samples into the initial model, and obtain multi-scale features through the neural network, and obtain local descriptors;

[0024] 2-1. Add a multi-scale feature fusion module to the last four attention layers of the DINOv2 network to construct a neural network, and connect the multi-scale feature fusion module in parallel with the multi-layer perceptron in the last four attention layers of the DINOv2 network;

[0025] 2-2. Input the preprocessed training sample into the neural network and pass through the tail of the last four layers to obtain feature X′ n :

[0026] X′ n =MHA(LN(X n-1 ))+X n-1 ;

[0027] Among them, MHA() is the multi-head self-attention mechanism in the attention layer, LN() is the layer normalization method, and X n-1 is the feature obtained by the previous attention layer, and n is the number of attention layers;

[0028] For the feature X′ n After downsampling to 1 / 2 of the dimension and passing through the activation function ReLU, the reduced-dimensional and activated feature X′ is obtained n ;

[0029] 2-3. The dimension-reduced and activated feature X′ n The intermediate features are obtained through a 1×1 convolution layer to increase the nonlinear features;

[0030] 2-4, the intermediate features are respectively subjected to 1×1, 3×3, and 5×5 convolution layers to obtain features of 1×1 scale, 3×3 scale, and 5×5 scale, respectively;

[0031] 2-5. The features of the 1×1 scale, 3×3 scale, and 5×5 scale are further subjected to a 1×1 convolution layer to change the number of channels of the intermediate features to obtain features with changed channel numbers, so as to achieve cross-channel information interaction;

[0032] 2-6. After the features with changed channel numbers are upsampled and restored to the original dimensions, a local descriptor X with multi-scale features is obtained. n :

[0033] X n =MLP(LN(X′ n ))+s·MFF(LN(X′ n ))+X′ n ;

[0034] Among them, MLPO is a multi-layer perceptron, LN() is a layer normalization method, MFFO is a multi-layer feature fusion module, s is a scaling factor, and n is the number of attention layers;

[0035] At the same time, get the global token.

[0036] Step 3: for the local descriptors with multi-scale features, cluster the local descriptors using the optimal transmission algorithm and aggregate the global descriptors with the aid of registers, train the neural network, optimize the neural network parameters, and obtain a trained model;

[0037] 3-1. Initialize k registers of the same size as the local descriptor with multi-scale features;

[0038] 3-2. After fractional projection, the local descriptor with multi-scale features is dimensionally reduced together with the initialized register to obtain a dimensionally reduced local descriptor and a dimensionally reduced register;

[0039] 3-3. The local descriptors after dimension reduction and the registers after dimension reduction are first subjected to an optimal transmission algorithm to re-cluster the features to the clusters, thereby obtaining re-clustered local features and re-clustered registers;

[0040] 3-4. Construct two attention encoding layers. Each attention encoding layer consists of layer normalization, multi-head self-attention mechanism, and multi-layer perceptron.

[0041] 3-5. Input the re-clustered local features and the re-clustered registers into the two attention encoding layers to obtain the re-weighted local features and the re-weighted registers;

[0042] 3-6. Discard the registers after the reallocation of weights, and only retain the local features after the reallocation of weights;

[0043] 3-7. Input the global token into the score projection to obtain a projected global token;

[0044] 3-8. Combining the projected global tokens and the local descriptors after reallocation of weights to form a global descriptor for searching and matching;

[0045] 3-9. Use the adaptive learning rate optimization algorithm Adam to calculate the gradient and update the parameters of the neural network and the weights of the two attention encoding layers to obtain the trained model.

[0046] Step 4: input the test image in the test sample into the trained model to obtain a verification image matching the test sample, and output the recall rate;

[0047] 4-1. Inputting the test image in the test sample into the trained model and extracting the global descriptor of the test image;

[0048] 4-2. Determine the matching result of the test image and output the recall rate Recall@k, where k is 1, 5 and 10. Recal1@1 indicates the percentage of correct retrieval times in the first-ranked sequence of retrieval results to the total retrieval times. Recall@5 indicates the percentage of correct retrieval times in the top five retrieval result sequences to the total retrieval times. Recall@10 indicates the percentage of correct retrieval times in the top ten retrieval result sequences to the total retrieval times.

[0049] The trained network is used to test the test data set to obtain indicators and visualization results, and compared with the latest SALAD method. The comparison of the recall rate indicator between the present invention and the famous method SALAD shows that the present invention exceeds the SALAD method in terms of recall rate.

[0050] Combination Figure 2 ,Comparison of the present invention and SALAD in attention heatmap, the first column is the original query graph, the second column is the attention heatmap obtained by the SALAD method, and the third column is the attention heatmap obtained by the present invention. It can be seen that the SALAD method discards some features with robust representation, while the present invention successfully retains these features such as buildings, and does not pay too much attention to global background features and dynamic non-robust features such as cars, which improves the retrieval accuracy of the model.

[0051] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0052] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A register-assisted aggregated visual location recognition method based on DINOv2, characterized in that: The following steps are involved: Step 1: Collect street view images in the data set to form training samples and test samples, and pre-process the training samples and the test samples; Step 2: construct a neural network including a multi-scale feature fusion module based on the DINOv2 network; put the preprocessed training samples into the initial model, and obtain multi-scale features through the neural network, and obtain local descriptors; Step 3: for the local descriptors with multi-scale features, cluster the local descriptors using the optimal transmission algorithm and aggregate the global descriptors with the aid of registers, train the neural network, optimize the neural network parameters, and obtain a trained model; Step 4: Input the test image in the test sample into the trained model to obtain a verification image matching the test sample, and output the recall rate.

2. According to claim 1, a register-assisted aggregated visual position recognition method based on DINOv2 is characterized in that: The step 1 specifically comprises the following steps: 1-1. Collect street view images from GSV-Cities, MSLS, Pittsburgh250k, Pittsburgh30k, NordLand, and SPED datasets, use the GSV-Cities dataset as training samples, and use the other datasets as test samples; 1-2. Preprocessing the training samples and the test samples, wherein the preprocessing adjusts the resolution of the street view image to 224×224 by a bilinear interpolation method.

3. The register-assisted aggregated visual position recognition method based on DINOv2 according to claim 1 is characterized in that: The step 2 specifically includes the following steps: 2-1. Add a multi-scale feature fusion module to the last four attention layers of the DINOv2 network to construct a neural network, and connect the multi-scale feature fusion module in parallel with the multi-layer perceptron in the last four attention layers of the DINOv2 network; 2-2. Input the preprocessed training sample into the neural network and pass through the tail of the last four layers to obtain feature X′ n : X′ n =MHA(LN(X n-1 ))+X n-1 ; Among them, MHA() is the multi-head self-attention mechanism in the attention layer, LN() is the layer normalization method, and X n-1 is the feature obtained by the previous attention layer, and n is the number of attention layers; For the feature X′ n After downsampling to 1 / 2 of the dimension and passing through the activation function ReLU, the reduced-dimensional and activated feature X′ is obtained n ; 2-3. The dimension-reduced and activated feature X′ n The intermediate features are obtained through a 1×1 convolution layer; 2-4, the intermediate features are respectively subjected to 1×1, 3×3, and 5×5 convolution layers to obtain features of 1×1 scale, 3×3 scale, and 5×5 scale, respectively; 2-5. The features of the 1×1 scale, 3×3 scale, and 5×5 scale are further subjected to a 1×1 convolution layer to change the number of channels of the intermediate features to obtain features with changed channel numbers; 2-6. After the features with changed channel numbers are upsampled and restored to the original dimensions, a local descriptor X with multi-scale features is obtained. n : X n =MLP(LN(X′ n ))+s·MFF(LN(X′ n ))+X′ n ; Among them, MLPO is a multi-layer perceptron, LN() is a layer normalization method, MFFO is a multi-layer feature fusion module, s is a scaling factor, and n is the number of attention layers; At the same time, get the global token.

4. The register-assisted aggregated visual position recognition method based on DINOv2 according to claim 1 is characterized in that: The step 3 specifically comprises the following steps: 3-1. Initialize k registers of the same size as the local descriptor with multi-scale features; 3-2. After fractional projection, the local descriptor with multi-scale features is dimensionally reduced together with the initialized register to obtain a dimensionally reduced local descriptor and a dimensionally reduced register; 3-3. The local descriptors after dimension reduction and the registers after dimension reduction are first subjected to an optimal transmission algorithm to re-cluster the features to the clusters, thereby obtaining re-clustered local features and re-clustered registers; 3-4. Construct two attention encoding layers. Each attention encoding layer consists of layer normalization, multi-head self-attention mechanism, and multi-layer perceptron. 3-5. Input the re-clustered local features and the re-clustered registers into the two attention encoding layers to obtain the re-weighted local features and the re-weighted registers; 3-6. Discard the registers after the reallocation of weights, and only retain the local features after the reallocation of weights; 3-7. Input the global token into the score projection to obtain a projected global token; 3-8. Combining the projected global tokens and the local descriptors after reallocation of weights to form a global descriptor for searching and matching; 3-9. Use the adaptive learning rate optimization algorithm Adam to calculate the gradient and update the parameters of the neural network and the weights of the two attention encoding layers to obtain the trained model.

5. The register-assisted aggregated visual position recognition method based on DINOv2 according to claim 1 is characterized in that: The step 4 specifically comprises the following steps: 4-1. Inputting the test image in the test sample into the trained model and extracting the global descriptor of the test image; 4-2. Determine the matching result of the test image and output the recall rate.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

7. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute any one of claims 1 to 5.

8. A computer program product, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 5.

Images