Image position identification method and device of basic vision model

Abstract

The invention discloses an image position identification method and device for a basic visual model, and the method employs the dynamic power of a first dynamic power normalization layer to enhance the features with rich background information, and weakens and even eliminates the features with less background information, thereby being capable of retaining more background information, and improving the image position identification efficiency. According to the method, the VPR based on background information identification can accurately identify the position of a query image, and the mass center-free detection module is used for aggregating a plurality of local enhancement description vectors and removing each mass center at the same time, so that the parameter quantity is reduced, the training is more efficient, and the influence of the mass center on the identification effect is avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image position recognition method and device of a basic visual model. Background Art

[0002] Visual Place Recognition (VPR) is crucial for mobile robots and autonomous vehicles because it is essential for full local positioning, simultaneous localization and mapping (SLAM), and scene understanding. VPR is usually viewed as an image retrieval problem, where the goal is to match a query image (an image representing the current location) with images of previously visited places. In order to achieve efficient matching, the Euclidean distance is usually used to represent the distance between image description vectors. Therefore, the VPR system first uses a backbone model to extract multiple local description vectors for each image and aggregate them into a global description vector.

[0003] However, compared with traditional image retrieval tasks, VPR faces unique challenges, including drastic changes in image viewpoints, seasonal variations, and occlusions. Therefore, many researchers are committed to exploring robust local description vectors that are invariant under these challenges. The emergence of deep learning has significantly promoted the development of VPR, making it more robust and more accurate. However, deep learning-based methods usually require training models from scratch on data from a specific environment. The diversity of application environments makes it difficult to collect enough data in different environments.

[0004] The paper "Izquierdo Sergio et al., Optimal Transport Aggregation for Visual Place Recognition" introduces Visual Foundation Models (VFM) to reduce the demand for training samples. However, the current research in this field is limited to the introduction of VFM, and has not explored in depth the model adapter and probing layer construction methods for the VPR field.

[0005] The most popular detection techniques currently include linear probing (LP) for classification, however, LP only uses simple first-order statistical features and its fine-tuning performance is limited. The NetVLAD method proposed in the literature "Arandjelovic Relja et al. NetVLAD: CNN architecture for weakly supervised place recognition" is widely used in the VPR field.

[0006] The invention patent application with publication number CN114241227A discloses a method and device for image recognition based on VLAD, the method comprising: obtaining an image to be classified; inputting the image to be classified into a trained image classification model, the model comprising an improved AlexNet module, a VLAD module, an LDA dimension reduction module and an SVM classification module; obtaining a local feature descriptor of the image to be classified based on the AlexNet module; inputting the local feature descriptor into the VLAD module based on the NetVLAD layer in the improved AlexNet module to obtain a feature vector of the image to be classified; inputting the feature vector into the LDA dimension reduction module to obtain a final vector representation of the image to be classified; inputting the final vector representation into the SVM classification module to obtain a classification result of the image to be classified. However, the NetVLAD disclosed in the above document requires expensive offline initialization of semantic centroids, which limits its flexibility in fine-tuning on different data sets. In addition, inaccurate centroids may introduce inductive biases (for example, initializing centroids in urban scenes, but training or reasoning in rural scenes), thereby affecting the generalization ability of the model. On the other hand, semantic centroids serve as prior knowledge for detection (aggregation), and simply removing them may lead to degraded aggregation performance. Summary of the invention

[0007] The present invention provides an image position recognition method of a basic visual model. The method enhances the recognition ability of the model by inserting a dynamic power normalization layer to enhance background information, and can also improve the generalization ability of the model by non-centroid detection.

[0008] A specific embodiment of the present invention provides an image position recognition method of a basic visual model, comprising:

[0009] The obtained pictures are used as training samples, and the real IDs of the obtained pictures are used as labels. A training sample set is constructed by using multiple training samples.

[0010] Constructing a training model, the training model includes an improved ViT module and a centroid-free detection module, dividing a training sample into multiple image blocks, inputting the multiple image blocks into the improved ViT module to obtain multiple local enhanced description vectors, the improved ViT module includes a first dynamic power normalization layer, the first dynamic power normalization layer is connected to the fully connected layer of the ViT module, and is used to adjust the feature vector output by the fully connected layer based on the L2 norm constructed by the dynamic power to enhance the background information feature, the dynamic power is obtained by linearly transforming the feature output of the fully connected layer after average pooling, the centroid-free detection module includes a second dynamic power normalization layer and two linear transformation layers, one of which includes a Sigmoid function, multiple local enhanced description vectors are input into the two linear transformation layers for dimension compression and then fused, and the fusion result is feature enhanced by the second dynamic power normalization layer to obtain a global vector;

[0011] A loss function is constructed based on the predicted ID and label corresponding to the global vector, and the first dynamic power normalization layer and the centroid-free detection module are trained using the loss function through the training sample set to obtain the basic visual model;

[0012] When applied, the image to be identified is input into the basic visual model to obtain a predicted global vector, and the predicted global vector is distance matched with the global vector of the image in the database to obtain the position of the image to be identified.

[0013] Preferably, multiple local enhancement description vectors are input into two linear transformation layers for dimension compression and then fused, including:

[0014] The two linear transformation layers include a first linear transformation layer and a second linear transformation layer;

[0015] The first linear transformation layer is a first feature transformation layer composed of two linear layers, and the dimensions of the multiple local enhancement description vectors are compressed by the first feature transformation layer to obtain a first compressed vector;

[0016] The second linear transformation layer includes a second feature conversion layer composed of two linear layers and a Sigmoid function, and multiple local enhancement description vectors are sequentially input into the second feature conversion layer and the Sigmoid function to obtain a second compression vector;

[0017] The first compressed vector is multiplied by the second compressed vector to complete the vector fusion.

[0018] Preferably, the dimensions of the multiple local enhancement description vectors are compressed by the first feature conversion layer to obtain a first compressed vector, wherein the first compressed vector for:

[0019]

[0020] Among them, Fea is a number of local enhancement description vectors, Linear represents the linear layer, and Relu is the activation function.

[0021] Preferably, each element of the feature vector output by the fully connected layer is divided by the L2 norm constructed by the dynamic power to adjust the feature vector output by the fully connected layer, and the L2 norm Y constructed by the dynamic power is:

[0022] Y=||sign(X)|X| a ||2

[0023] Among them, sign(﹒) is the sign function, X is the feature vector output by the fully connected layer, α is the dynamic power, and ||·||2 is the L2 norm.

[0024] Preferably, the dynamic power is obtained by performing linear transformation after average pooling of the features output by the fully connected layer, and the dynamic power α is:

[0025] α=Sigmoid(Linear(Relu(Linear(Avg_pool(X)))))

[0026] Among them, X is the feature vector output by the fully connected layer, Linear represents the linear layer, Relu is the activation function, and Avg_pool represents average pooling.

[0027] Preferably, the improved ViT module includes a plurality of fine-tuning submodules, and each local vector is passed through the plurality of fine-tuning submodules to obtain a corresponding local enhanced description vector;

[0028] Each fine-tuning submodule includes a first normalization layer, a multi-head attention layer, a second normalization layer, a fully connected layer, and a first dynamic power normalization layer;

[0029] Among them, each local vector is input into the first normalization layer and the multi-head attention layer in turn to obtain the attention feature, the attention feature and the local vector are summed for the first time to obtain the first sum feature, the first sum feature is sequentially passed through the second normalization layer, the fully connected layer and the first dynamic power normalization layer to obtain the enhanced feature, and the enhanced feature and the first sum feature are summed for the second time to obtain the second sum feature.

[0030] Preferably, the training sample is divided into a plurality of picture blocks, including:

[0031] The training samples are cut into block images of fixed size, and each block image is input into a projection layer composed of a fully connected layer, so that each block image is projected into a corresponding local vector.

[0032] The present invention also provides an image position recognition device for a basic visual model, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the image position recognition method for a basic visual model.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention utilizes the dynamic power of the first dynamic power normalization layer to enhance features with rich background information and weaken or even eliminate features with less background information, thereby allowing the global vector to retain more background information, so that the VPR that relies on background information recognition can more accurately identify the position of the query image.

[0035] The present invention also utilizes a centroid-free detection module to aggregate multiple local enhanced description vectors, and at the same time, removes the centroid to reduce the number of parameters, improve training efficiency, and avoid the negative impact of the centroid on the recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of an image position recognition method of a basic visual model provided by a specific embodiment of the present invention;

[0037] Figure 2 A flowchart of an image position recognition method of a basic visual model provided in a specific embodiment of the present invention;

[0038] Figure 3 A schematic diagram of the structure of an image position recognition device based on a basic visual model provided in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The specific embodiment of the present invention introduces a new centroid-free probing (CFP) technology to avoid explicit calculation of semantic centroids, thereby avoiding the introduction of bias, and achieving the flexibility of fine-tuning the training set. In addition, since VPR mainly relies on small overlapping areas between different images to make judgments when dealing with changes in image perspective, it is crucial to retain the information of these distinguishing areas. In addition, due to the difference in training objectives, that is, the general representation capabilities of pre-trained VFMs often focus more on foreground objects, while VPR tasks rely more on background areas such as buildings. The present application inserts a dynamic power normalization (DPN) layer into the backbone network to enhance the retention of information in these key background areas. In the case of a frozen backbone network, the method provided by the present application can achieve excellent performance only through training with adapters and centroid-free probing. The specific embodiment of the present invention provides an image position recognition method for a basic visual model, Figure 1 and Figure 2 As shown, including:

[0041] (1) The obtained images are used as training samples, and the real IDs of the obtained images are used as labels. Multiple training samples are used to construct a training sample set.

[0042] (2) Construct a training model, which includes a Patchify module, an improved ViT module and a centroid-free detection module.

[0043] The specific embodiment of the present invention divides the training samples into multiple local vectors through the Patchify module. In a specific embodiment, the training samples are discretized through the Patchify module, so that the images can be processed by the Transformer model like text. The specific embodiment of the present invention needs to cut the input image into patches of fixed size (such as 14*14), and the total number of patches is L. Each patch image is output to the projection layer composed of the fully connected layer, and each patch image is projected into a vector of length C.

[0044] A specific embodiment of the present invention inputs multiple local vectors into an improved ViT module to obtain multiple local enhanced description vectors. The improved ViT module includes N fine-tuning sub-modules (blocks). Each local vector is passed through the N fine-tuning sub-modules to obtain a corresponding local enhanced description vector. The Fea provided by the specific embodiment of the present invention consists of L local enhanced description vectors with a length of C. It is called a local enhanced description vector because the receptive field represented by each vector only occupies a small area of ​​the corresponding image.

[0045] Each fine-tuning submodule includes a first normalization layer (LayersNorm), a multi-head attention layer, a second normalization layer (LayersNorm), a fully connected layer (Multilayer Perceptron, MLP) and a first dynamic power normalization layer (Dynamic Power Normalization, DPN). Each local vector is sequentially input into the first linear layer and the multi-head attention layer to obtain the attention feature, the attention feature and the local vector are first summed to obtain the first sum feature, the first sum feature is sequentially passed through the second linear layer, the fully connected layer and the first dynamic power normalization layer to obtain the enhanced feature, and the enhanced feature and the first sum feature are second summed to obtain the second sum feature.

[0046] The first dynamic power normalization layer provided in the specific embodiment of the present invention is connected to the fully connected layer, and is used to adjust the feature vector output by the fully connected layer based on the L2 norm constructed by the dynamic power to enhance the background information characteristics. The dynamic power is obtained by linearly transforming the features output by the fully connected layer after average pooling. The dynamic power is a scalar. The dynamic power provided in the specific embodiment of the present invention is obtained based on the feature prediction of the fully connected layer output. Therefore, it can reflect the richness of the background information contained in the feature. If the task-related information in the picture is scarce, the corresponding features are weakened or even eliminated based on the dynamic power. If the information is rich, the corresponding features are enhanced based on the dynamic power.

[0047] In a specific embodiment of the present invention, each element in the feature vector output by the fully connected layer is divided by the L2 norm constructed by the dynamic power to adjust the feature vector output by the fully connected layer. The L2 norm Y constructed by the dynamic power is:

[0048] Y=||sign(X)|X| a ||2

[0049] Among them, sign(﹒) is the sign function, X is the feature vector output by the fully connected layer, α is the dynamic power, and ||·||2 is the L2 norm.

[0050] The dynamic power α provided in the specific embodiment of the present invention is:

[0051] α=Sigmoid(Linear(Relu(Linear(Avg_pool(X)))))

[0052] Among them, X is the feature vector output by the fully connected layer, Linear represents the linear layer, Sigmoid and Relu are two different activation functions, and Avg_pool represents average pooling.

[0053] The centroid-free detection module provided in a specific embodiment of the present invention includes a second dynamic power normalization layer and two linear transformation layers, wherein one linear transformation layer includes a Sigmoid function for eliminating the centroid, and multiple local enhancement description vectors are input into the two linear transformation layers for dimension compression and then fused, and the fusion result is feature enhanced by the second dynamic power normalization layer to obtain a global vector.

[0054] The fusion result G provided by the specific embodiment of the present invention is:

[0055]

[0056] in, It is the first feature conversion layer, which consists of two linear layers. Multiple local enhancement description vectors Fea are input into the first feature conversion layer to obtain the first compressed vector. The size of the first compressed vector is L*D, where D is a value less than C. Therefore, the latitude of the feature can be compressed to save feature storage space. The second feature conversion layer has the same architecture as the first feature conversion layer, but the size of the compressed vector obtained by inputting multiple local enhancement description vectors Fea into the second feature conversion layer is L*K. The compressed vector is subjected to sigmoid operation by column through the Sigmoid function, so that the weight sum of each implicit centroid assigned to the local vector is always 1. This prevents a certain implicit centroid from having too much or too little influence on the training. This reduces the number of parameters, speeds up the training, and avoids the influence of the centroid on the model training.

[0057] In one embodiment, the first compressed vector for:

[0058]

[0059] Among them, Fea is a number of local enhancement description vectors, Linear represents the linear layer, and Relu is the activation function.

[0060] The specific embodiment of the present invention uses the DPN layer again to perform feature enhancement to obtain the global vector The global vector for:

[0061]

[0062] b=Sigmoid(Linear(Relu(Linear(Avg_pool(G))))).

[0063] (3) A loss function is constructed based on the predicted ID and label corresponding to the global vector. The first dynamic power normalization layer and the centroid-free detection module are trained using the loss function through the training sample set to obtain a basic visual model. The present invention only trains the first dynamic power normalization layer and the centroid-free detection module. Since the number of parameters of these two modules is relatively small, the training is efficient.

[0064] (4) When applied, the image to be identified is input into the basic visual model to obtain a predicted global vector, and the predicted global vector is distance-matched with the global vector of the image in the database, and the ID of the image with the best distance match is used as the position of the image to be identified.

[0065] like Figure 3 As shown, a specific embodiment of the present invention further provides an image position recognition device of a basic visual model, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the image position recognition method of a basic visual model.

Claims

1. A method for image position recognition based on a basic visual model, characterized in that: include: The obtained pictures are used as training samples, and the real IDs of the obtained pictures are used as labels. A training sample set is constructed by using multiple training samples. Constructing a training model, the training model includes an improved ViT module and a centroid-free detection module, dividing a training sample into multiple image blocks, inputting the multiple image blocks into the improved ViT module to obtain multiple local enhanced description vectors, the improved ViT module includes a first dynamic power normalization layer, the first dynamic power normalization layer is connected to the fully connected layer of the ViT module, and is used to adjust the feature vector output by the fully connected layer based on the L2 norm constructed by the dynamic power to enhance the background information feature, the dynamic power is obtained by linearly transforming the feature output of the fully connected layer after average pooling, the centroid-free detection module includes a second dynamic power normalization layer and two linear transformation layers, one of which includes a Sigmoid function, multiple local enhanced description vectors are input into the two linear transformation layers for dimension compression and then fused, and the fusion result is feature enhanced by the second dynamic power normalization layer to obtain a global vector; A loss function is constructed based on the predicted ID and label corresponding to the global vector, and the first dynamic power normalization layer and the centroid-free detection module are trained using the loss function through the training sample set to obtain the basic visual model; When applied, the image to be identified is input into the basic visual model to obtain a predicted global vector, and the predicted global vector is distance matched with the global vector of the image in the database to obtain the position of the image to be identified.

2. The image position recognition method of the basic visual model according to claim 1 is characterized in that: Multiple local enhancement description vectors are input into two linear transformation layers for dimension compression and fusion, including: The two linear transformation layers include a first linear transformation layer and a second linear transformation layer; The first linear transformation layer is a first feature transformation layer composed of two linear layers, and the dimensions of the multiple local enhancement description vectors are compressed by the first feature transformation layer to obtain a first compressed vector; The second linear transformation layer includes a second feature conversion layer composed of two linear layers and a Sigmoid function, and multiple local enhancement description vectors are sequentially input into the second feature conversion layer and the Sigmoid function to obtain a second compression vector; The first compressed vector is multiplied by the second compressed vector to complete the vector fusion.

3. The image position recognition method of the basic visual model according to claim 2 is characterized in that: The dimensions of the multiple local enhancement description vectors are compressed by the first feature conversion layer to obtain a first compressed vector. for: Among them, Fea is a number of local enhancement description vectors, Linear(﹒) represents the linear layer, and Relu(﹒) is the activation function.

4. The image position recognition method of the basic visual model according to claim 1 is characterized in that: Each element of the feature vector output by the fully connected layer is divided by the L2 norm constructed by the dynamic power to adjust the feature vector output by the fully connected layer. The L2 norm Y constructed by the dynamic power is: Y=||sign(X)|X| a ||2 Among them, sign(﹒) is the sign function, X is the feature vector output by the fully connected layer, α is the dynamic power, and ||·||2 is the L2 norm.

5. The image position recognition method of the basic visual model according to claim 4 is characterized in that: The dynamic power is obtained by performing linear transformation after average pooling of the features output by the fully connected layer. The dynamic power α is: α=Sigmoid(Linear(Relu(Linear(Avg_pool(X))))) Among them, X is the feature vector output by the fully connected layer, Linear(﹒) represents the linear layer, Relu(﹒) is the activation function, and Avg_pool(﹒) represents average pooling.

6. The image position recognition method of the basic visual model according to claim 1, characterized in that: The improved ViT module includes a plurality of fine-tuning submodules, and each local vector is passed through the plurality of fine-tuning submodules to obtain a corresponding local enhanced description vector; Each fine-tuning submodule includes a first normalization layer, a multi-head attention layer, a second normalization layer, a fully connected layer, and a first dynamic power normalization layer; Among them, each local vector is input into the first normalization layer and the multi-head attention layer in turn to obtain the attention feature, the attention feature and the local vector are summed for the first time to obtain the first sum feature, the first sum feature is sequentially passed through the second normalization layer, the fully connected layer and the first dynamic power normalization layer to obtain the enhanced feature, and the enhanced feature and the first sum feature are summed for the second time to obtain the second sum feature.

7. The image position recognition method of the basic visual model according to claim 1 is characterized in that: The training samples are divided into multiple image blocks, including: The training samples are cut into block images of fixed size, and each block image is input into a projection layer composed of a fully connected layer, so that each block image is projected into a corresponding local vector.

8. An image position recognition device based on a basic visual model, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement an image position recognition method of a basic visual model described in any one of claims 1-7.

Images