A register-assisted aggregated visual position recognition method based on DINOv2

Abstract

The application discloses a register-assisted aggregated visual position recognition method based on DINOv2, which comprises the following steps: step 1, collecting street view images in a data set to form training samples and test samples, and preprocessing the training samples and the test samples; step 2, constructing a neural network containing a multi-scale feature fusion module according to a DINOv2 network; putting the preprocessed training samples into an initial model, and obtaining multi-scale features and local descriptors through the neural network; step 3, training the neural network, optimizing the neural network parameters, and obtaining a trained model; step 4, inputting test images in the test samples into the trained model to obtain verification images matched with the test samples, and outputting a recall rate. The application enhances the representation ability of features under different resolutions, improves the robustness of global descriptors, and improves the accuracy of position retrieval.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a register-assisted aggregation visual position recognition method based on DINOv2. BACKGROUND

[0002] Visual position recognition (VPR) aims to retrieve the most matching image from a database of visual scenes containing geographic position labels to estimate the position information of the current query image. VPR has been widely used in mobile robots and augmented reality for a long time, such as automatic driving, image geolocation and three-dimensional reconstruction. The main challenges include conditional changes such as light and weather and season, viewpoint changes, perceptual aliasing and appearance changes over time.

[0003] A recent work, Sinkhorn Algorithm for Local Aggregation of Descriptors (SALAD), more efficiently assigns local features to global descriptors by re-modeling the feature-to-cluster assignment problem of the optimal transport algorithm. To further improve the discriminability of the global descriptor, SALAD discards uninformative features by introducing a "trash can" mechanism in the aggregation process. However, by extracting and analyzing the heat map results of the SALAD retrieval image, it is found that the "trash can" mechanism in SALAD easily discards part of the robust information such as buildings as useless information, while still retaining part of the dynamic non-robust information such as cars. Such a "trash can" mechanism reduces the feature representation ability of the model, thereby reducing the retrieval performance of the model. SUMMARY

[0004] The application aims to provide a register-assisted aggregation visual position recognition method based on DINOv2, which enhances the feature representation ability under different resolutions, improves the robustness of the global descriptor, and improves the accuracy of position retrieval.

[0005] In order to achieve the purpose of the application, the application provides a register-assisted aggregation visual position recognition method based on DINOv2, comprising the following steps:

[0006] Step 1, collect street view images in the data set to form training samples and test samples, and pretreat the training samples and the test samples;

[0007] Step 2, constructing a neural network containing a multi-scale feature fusion module according to the DINOv2 network; putting the pretreated training samples into an initial model, and obtaining multi-scale features through the neural network, and obtaining local descriptors at the same time;

[0008] Step 3, clustering the local descriptors and register-assisted aggregating the global descriptors using the optimal transport algorithm for the local descriptors with multi-scale features, training the neural network, optimizing the neural network parameters, and obtaining a trained model;

[0009] Step 4, inputting a test image in the test sample into the trained model to obtain a verification image matched with the test sample, and outputting a recall rate.

[0010] An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the above-mentioned register-assisted aggregation visual position recognition method based on DINOv2 when executing the program.

[0011] A non-transitory computer-readable storage medium, characterized in that the non-transitory computer-readable storage medium stores computer instructions for causing the computer to execute the above-mentioned register-assisted aggregation visual position recognition method based on DINOv2.

[0012] A computer program product, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the above-mentioned register-assisted aggregation visual position recognition method based on DINOv2.

[0013] Compared with the prior art, the significant progress of the present application is that: (1) the present application proposes a multi-scale feature fusion module, which enhances the representation ability of features at different resolutions by fusing the multi-scale features of the input image; (2) the present application proposes a register-assisted aggregation module to filter out global background features and dynamic non-robust features in the image, thereby effectively improving the robustness of the global descriptor and improving the position retrieval accuracy; (3) the experimental results on 5 widely used datasets MSLS, Pittsburgh250k, Pittsburgh30k, NordLand and SPED show that RegVPR is superior to SOTA method.

[0014] To more clearly illustrate the functional characteristics and structural parameters of the present application, the following further describes the present application in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0015] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 is the overall architecture of the present application;

[0017] Figure 2 is the attention result of the present application compared with the SALAD method. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0019] A register-assisted aggregated visual location recognition method based on DINOv2 of the present application combines Figure 1 , comprising the following steps:

[0020] Step 1, collecting street view images in a data set to form training samples and test samples, and preprocessing the training samples and the test samples;

[0021] 1-1, collecting street view images in GSV-Cities, MSLS, Pittsburgh250k, Pittsburgh30k, NordLand and SPED data sets, taking the GSV-Cities data set as the training samples and taking the other data sets as the test samples;

[0022] 1-2, preprocessing the training samples and the test samples, and adjusting the resolution of the street view images to 224x224 by a bilinear interpolation method.

[0023] Step 2, constructing a neural network containing a multi-scale feature fusion module according to a DINOv2 network; putting the preprocessed training samples into an initial model and obtaining multi-scale features and local descriptors through the neural network;

[0024] 2-1, constructing a neural network by adding a multi-scale feature fusion module in the last four attention layers of the DINOv2 network, and connecting the multi-scale feature fusion module and the multi-layer perceptron in the last four attention layers of the DINOv2 network in parallel;

[0025] 2-2, inputting the preprocessed training samples into the neural network, passing through the tail of the last four layers to obtain feature X' n :

[0026] X′ n = MHA(LN(X n-1 ))+X n-1 ;

[0027] wherein MHA() is a multi-head self-attention mechanism in the attention layer, LN() is a layer normalization method, X n-1 is a feature obtained by a previous attention layer, and n is the number of attention layers.

[0028] The feature X′ n is down-sampled to 1 / 2 of the dimension and passed through an activation function ReLU to obtain a reduced and activated feature X′ n .

[0029] 2-3, the reduced and activated feature X′ n is passed through a 1x1 convolution layer to obtain an intermediate feature so as to increase nonlinear features.

[0030] 2-4, the intermediate feature is respectively passed through a 1x1, 3x3 and 5x5 convolution layer to obtain a 1x1 scale, 3x3 scale and 5x5 scale feature.

[0031] 2-5, the 1x1 scale, 3x3 scale and 5x5 scale feature is further passed through a 1x1 convolution layer to change the channel number of the intermediate feature to obtain a feature with changed channel number so as to realize cross-channel information interaction.

[0032] 2-6, the feature with changed channel number is up-sampled to restore to the original dimension to obtain a local descriptor X n with multi-scale features.

[0033] X n = MLP(LN(X′ n ))+s·MFF(LN(X′ n ))+X′ n .

[0034] wherein MLP0 is a multi-layer perceptron, LN() is a layer normalization method, MFF0 is a multi-layer feature fusion module, s is a scaling factor, and n is the number of attention layers.

[0035] A global token is obtained at the same time.

[0036] Step 3, the local descriptor with multi-scale features is clustered by using an optimal transport algorithm, and a global descriptor is aggregated with register assistance, the neural network is trained, the neural network parameters are optimized, and a trained model is obtained.

[0037] 3-1, initializing k registers with the same size as the local descriptor with multi-scale features.

[0038] 3-2, the local descriptor with multi-scale features is dimensionally reduced after fractional projection, and the initialized registers are dimensionally reduced to obtain a reduced local descriptor and a reduced register.

[0039] 3-3、The reduced dimension local descriptor and the reduced dimension register are first subjected to an optimal transmission algorithm, and the allocation of the re-clustered features to the clusters is obtained, thereby obtaining the re-clustered local features and the re-clustered register;

[0040] 3-4、Two layers of attention encoding layers are constructed, and each layer of attention encoding layer is composed of layer normalization, multi-head self-attention mechanism and multi-layer perceptron;

[0041] 3-5、The re-clustered local features and the re-clustered register are input into the two layers of attention encoding layers, thereby obtaining the re-allocated weight local features and the re-allocated weight register;

[0042] 3-6、The re-allocated weight register is discarded, and only the re-allocated weight local features are retained;

[0043] 3-7、The global token is input into a fractional projection, thereby obtaining the projected global token;

[0044] 3-8、The projected global token and the re-allocated weight local descriptor are combined, and are aggregated into a global descriptor used for searching and matching;

[0045] 3-9、An adaptive learning rate optimization algorithm Adam is used to calculate the gradient and update the parameters of the neural network and the weights of the two layers of attention encoding layers, thereby obtaining a trained model.

[0046] Step 4、The test image in the test sample is input into the trained model, thereby obtaining a verification image matched with the test sample, and outputting a recall rate;

[0047] 4-1、The test image in the test sample is input into the trained model, and the global descriptor of the test image is extracted;

[0048] 4-2、The matching result of the test image is determined, and a recall rate Recall@k is output, k is 1, 5 and 10, Recal1@1 represents the percentage of the number of correct retrievals in the total retrieval number in the sequence of the first retrieval result, Recall@5 represents the percentage of the number of correct retrievals in the total retrieval number in the sequence of the top five retrieval results, and Recall@10 represents the percentage of the number of correct retrievals in the total retrieval number in the sequence of the top ten retrieval results.

[0049] According to the trained network, the test data set is tested to obtain the index and visual result, and is compared with the latest SALAD method. According to the comparison of the recall rate index between the present application and the famous method SALAD, it can be seen that the present application exceeds the SALAD method in the recall rate.

[0050] In combination Figure 2 , the comparison between the present application and SALAD in the attention heat map, the first column is the original query graph, the second column is the attention heat map obtained by the SALAD method, and the third column is the attention heat map obtained by the present application. It can be seen that the SALAD method discards part of the features with robust representation, while the present application successfully retains these features such as buildings, and does not pay too much attention to global background features and dynamic non-robust features such as cars, thereby improving the retrieval accuracy of the model.

[0051] It should be noted that in this paper, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment.

[0052] Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A DINOv2-based register-assisted aggregated visual position recognition method, characterized in that, The method comprises the following steps: Step 1, collecting street view images in a data set to form training samples and test samples, and preprocessing the training samples and the test samples; Step 2, constructing a neural network comprising a multi-scale feature fusion module according to a DINOv2 network; placing the preprocessed training samples into an initial model and obtaining multi-scale features through the neural network, while obtaining local descriptors; 2-1, adding a multi-scale feature fusion module to the last four attention layers of the DINOv2 network to construct a neural network, and connecting the multi-scale feature fusion module in parallel with the multi-layer perceptron in the last four attention layers of the DINOv2 network; 2-2, input the pre-processed training sample into the neural network, pass through the tail of the last four layers, and obtain the feature X' n : X′ n = MHA(LN(X n-1 ))+X n-1 ; where MHA() is a multi-head self-attention mechanism in the attention layer, LN() is a layer normalization method, X n-1 is the feature obtained by the previous attention layer, and n is the number of attention layers. For the feature X' n After down-sampling to 1 / 2 of the dimension, and passing through the activation function ReLU, the reduced and activated feature X' is obtained n ; 2-3, the reduced and activated features X' n intermediate features are obtained through a 1x1 convolutional layer; 2-4, the intermediate features pass through 1×1, 3×3 and 5×5 convolution layers respectively to obtain 1×1 scale, 3×3 scale and 5×5 scale features respectively; 2-5, the 1×1 scale, 3×3 scale and 5×5 scale features pass through a 1×1 convolution layer again to change the channel number of the intermediate features to obtain features with changed channel numbers; 2-6、The feature of changing the number of channels is upsampled to restore the original dimension to obtain a local descriptor X with multi-scale features n : X n =MLP(LN(X′ n ))+s·MFF(LN(X′ n ))+X′ n ; wherein MLP() is a multi-layer perceptron, LN() is a layer normalization method, MFF() is a multi-layer feature fusion module, s is a scaling factor, and n is the number of attention layers; global tokens are obtained at the same time; Step 3, clustering local descriptors and register auxiliary aggregating global descriptors using an optimal transport algorithm for the local descriptors with multi-scale features, training the neural network, optimizing the neural network parameters, and obtaining a trained model; Step 4, inputting test images in the test samples into the trained model to obtain verification images matched with the test samples, and outputting a recall rate.

2. The DINOv2-based register-assisted aggregated visual position recognition method according to claim 1, characterized in that, The step 1 specifically comprises the following steps: 1-1, collecting street view images in GSV-Cities, MSLS, Pittsburgh250k, Pittsburgh30k, NordLand and SPED data sets, taking the GSV-Cities data set as training samples and taking other data sets as test samples; 1-2, preprocessing the training samples and the test samples, wherein the preprocessing adjusts the resolution of the street view images to 224×224 through a bilinear interpolation method.

3. The DINOv2-based register-assisted aggregated visual position identification method according to claim 1, wherein, The step 3 specifically comprises the following steps: 3-1, initializing k registers with the same size as the local descriptors with multi-scale features; 3-2, after the local descriptors with multi-scale features are projected by a fraction, the initialized registers are dimensionally reduced together with the local descriptors to obtain dimensionally reduced local descriptors and dimensionally reduced registers; 3-3, the dimensionally reduced local descriptors and the dimensionally reduced registers are first subjected to an optimal transport algorithm to recluster features to the allocation of clusters to obtain reclustered local features and reclustered registers; 3-4, constructing two attention encoding layers, each of which comprises a layer normalization, a multi-head self-attention mechanism and a multi-layer perceptron. 3-5, input the re-clustered local features and the re-clustered registers into the two-layer attention encoding layer to obtain re-allocated weight local features and re-allocated weight registers; 3-6, discard the re-allocated weight registers and only keep the re-allocated weight local features; 3-7, input the global token into fractional projection to obtain projected global token; 3-8, merge the projected global token and the re-allocated weight local features to aggregate into global descriptors used for searching matching; 3-9, use adaptive learning rate optimization algorithm Adam to calculate gradient and update parameters of the neural network and weights of the two-layer attention encoding layer to obtain trained model.

4. The DINOv2-based register-assisted aggregated visual position recognition method according to claim 1, characterized in that, The step 4 specifically comprises the following steps: 4-1, input the test image in the test sample into the trained model and extract global descriptors of the test image; 4-2, determine the matching result of the test image and output recall rate.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1 to 4 when executing the program.

6. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions for causing the computer to execute the method of any one of claims 1 to 4.

7. A computer program product comprising computer program instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 4.

Images