Image position recognition method and device based on basic visual model

Abstract

The present invention discloses a method and device for image position recognition of a basic visual model. The method disclosed by the present invention utilizes the dynamic power of a first dynamic power normalization layer to enhance features with rich background information and weaken or even eliminate features with less background information, thereby retaining more background information and enabling a VPR that relies on background information recognition to more accurately identify the position of a query image. The method disclosed by the present invention also utilizes a centroid-free detection module to aggregate multiple local enhancement description vectors while removing each centroid, thereby reducing the number of parameters, making training more efficient, and avoiding the influence of the centroid on the recognition effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image position recognition method and device based on a basic visual model. Background Art

[0002] Visual Place Recognition (VPR) is crucial for mobile robots and autonomous vehicles because it is essential for full localization, simultaneous localization and mapping (SLAM), and scene understanding. VPR is usually viewed as an image retrieval problem, where the goal is to match a query image (an image representing the current location) with images of previously visited places. In order to achieve efficient matching, the Euclidean distance is usually used to represent the distance between image description vectors. Therefore, the VPR system first uses a backbone model to extract multiple local description vectors for each image and aggregate them into a global description vector.

[0003] However, compared to traditional image retrieval tasks, VPR faces unique challenges, including drastic changes in image viewpoint, seasonal variations, and occlusions. Therefore, many researchers have devoted themselves to exploring robust local description vectors that are invariant to these challenges. The emergence of deep learning has significantly promoted the development of VPR, making it more robust and more accurate. However, deep learning-based methods typically require training models from scratch on data from specific environments. The diversity of application environments makes it difficult to collect sufficient data in different environments.

[0004] The paper "Izquierdo Sergio et al., Optimal Transport Aggregation for Visual Place Recognition" introduces Visual Foundation Models (VFMs) to reduce the need for training samples. However, current research in this field has only focused on the introduction of VFMs and has not delved into the construction of model adapters and probing layers specifically for VPR.

[0005] The most popular detection techniques currently available include linear probing (LP) for classification, which uses only simple first-order statistical features and has limited fine-tuning performance. This is followed by the NetVLAD method (Arandjelovic Relja et al., "NetVLAD: CNN architecture for weakly supervised place recognition"), which is widely used in the field of VPR.

[0006] The invention patent application with publication number CN114241227A discloses a VLAD-based image recognition method and device, comprising: obtaining an image to be classified; inputting the image to be classified into a trained image classification model, the model comprising an improved AlexNet module, a VLAD module, an LDA dimensionality reduction module, and an SVM classification module; obtaining a local feature descriptor of the image to be classified based on the AlexNet module; inputting the local feature descriptor into the VLAD module based on the NetVLAD layer in the improved AlexNet module to obtain a feature vector of the image to be classified; inputting the feature vector into the LDA dimensionality reduction module to obtain a final vector representation of the image to be classified; and inputting the final vector representation into the SVM classification module to obtain a classification result for the image to be classified. However, the NetVLAD disclosed in the above document requires expensive offline initialization of semantic centroids, which limits its flexibility in fine-tuning on different datasets. In addition, inaccurate centroids may introduce inductive bias (for example, initializing the centroid in an urban scene but performing training or inference in a rural scene), thereby affecting the generalization ability of the model. On the other hand, semantic centroids serve as prior knowledge for detection (aggregation), and simply removing them may lead to degraded aggregation performance. Summary of the Invention

[0007] The present invention provides an image position recognition method of a basic visual model. The method enhances background information by inserting a dynamic power normalization layer to strengthen the recognition ability of the model, and can also improve the generalization ability of the model through centroid-free detection.

[0008] A specific embodiment of the present invention provides an image position recognition method based on a basic visual model, comprising:

[0009] The obtained pictures are used as training samples, and the real IDs of the obtained pictures are used as labels. A training sample set is constructed by multiple training samples.

[0010] Constructing a training model, the training model includes an improved ViT module and a centroid-free detection module, dividing the training sample into multiple image blocks, inputting the multiple image blocks into the improved ViT module to obtain multiple local enhanced description vectors, the improved ViT module includes a first dynamic power normalization layer, the first dynamic power normalization layer is connected to the fully connected layer of the ViT module, and is used to adjust the feature vector output by the fully connected layer based on the L2 norm constructed by the dynamic power to enhance the background information feature, the dynamic power is obtained by averaging the features output by the fully connected layer and then performing a linear transformation, the no centroid detection module includes a second dynamic power normalization layer and two linear transformation layers, wherein one linear transformation layer includes a Sigmoid function, the multiple local enhanced description vectors are input into the two linear transformation layers for dimensional compression and then fused, and the fusion result is feature enhanced by the second dynamic power normalization layer to obtain a global vector;

[0011] A loss function is constructed based on the predicted ID and label corresponding to the global vector. The first dynamic power normalization layer and the centroid-free detection module are trained using the loss function through the training sample set to obtain the basic visual model.

[0012] When applied, the image to be identified is input into the basic visual model to obtain a predicted global vector, and the predicted global vector is distance-matched with the global vector of the image in the database to obtain the position of the image to be identified.

[0013] Preferably, multiple local enhancement description vectors are input into two linear transformation layers for dimension compression and then fused, including:

[0014] The two linear transformation layers include a first linear transformation layer and a second linear transformation layer;

[0015] The first linear transformation layer is a first feature transformation layer composed of two linear layers, and the dimensions of the multiple local enhancement description vectors are compressed by the first feature transformation layer to obtain a first compressed vector;

[0016] The second linear transformation layer includes a second feature conversion layer composed of two linear layers and a Sigmoid function, and the multiple local enhancement description vectors are sequentially input into the second feature conversion layer and the Sigmoid function to obtain a second compressed vector;

[0017] The first compressed vector is multiplied by the second compressed vector to complete vector fusion.

[0018] Preferably, the dimensions of the multiple local enhancement description vectors are compressed by the first feature conversion layer to obtain a first compressed vector. for:

[0019]

[0020] Among them, Fea is a number of local enhancement description vectors, Linear represents the linear layer, and Relu is the activation function.

[0021] Preferably, each element of the feature vector output by the fully connected layer is divided by the L2 norm constructed by the dynamic power to adjust the feature vector output by the fully connected layer. The L2 norm Y constructed by the dynamic power is:

[0022] Y=||sign(X)|X| a ||2

[0023] Where sign(﹒) is the sign function, X is the feature vector output by the fully connected layer, α is the dynamic power, and ||·||2 is the L2 norm.

[0024] Preferably, the dynamic power is obtained by performing linear transformation after average pooling of the features output by the fully connected layer, and the dynamic power α is:

[0025] α=Sigmoid(Linear(Relu(Linear(Avg_pool(X)))))

[0026] Among them, X is the feature vector output by the fully connected layer, Linear represents the linear layer, Relu is the activation function, and Avg_pool represents average pooling.

[0027] Preferably, the improved ViT module includes a plurality of fine-tuning submodules, and each local vector is passed through the plurality of fine-tuning submodules to obtain a corresponding local enhanced description vector;

[0028] Each fine-tuning submodule consists of a first normalization layer, a multi-head attention layer, a second normalization layer, a fully connected layer, and a first dynamic power normalization layer;

[0029] Among them, each local vector is input into the first normalization layer and the multi-head attention layer in turn to obtain the attention feature, the attention feature and the local vector are summed for the first time to obtain the first sum feature, the first sum feature is passed through the second normalization layer, the fully connected layer and the first dynamic power normalization layer in turn to obtain the enhanced feature, and the enhanced feature and the first sum feature are summed for the second time to obtain the second sum feature.

[0030] Preferably, the training sample is divided into a plurality of image blocks, including:

[0031] The training samples are cut into block images of fixed size, and each block image is input into a projection layer composed of a fully connected layer, so that each block image is projected into a corresponding local vector.

[0032] The present invention also provides an image position recognition device for a basic visual model, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the image position recognition method for a basic visual model.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention utilizes the dynamic power of the first dynamic power normalization layer to enhance features with rich background information and weaken or even eliminate features with less background information, thereby allowing the global vector to retain more background information, so that the VPR that relies on background information recognition can more accurately identify the position of the query image.

[0035] The present invention also utilizes a centroid-free detection module to aggregate multiple local enhancement description vectors, and at the same time, removes the centroid to reduce the number of parameters, improve training efficiency, and avoid the negative impact of the centroid on the recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of an image position recognition method based on a basic visual model provided in a specific embodiment of the present invention;

[0037] Figure 2 A flowchart of an image position recognition method based on a basic visual model provided in a specific embodiment of the present invention;

[0038] Figure 3 A schematic structural diagram of an image position recognition device based on a basic visual model provided in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] The specific embodiment of the present invention introduces a new centroid-free probing (CFP) technology to avoid explicit calculation of semantic centroids, thereby avoiding the introduction of bias, and at the same time achieving the flexibility of fine-tuning the training set. In addition, since VPR mainly relies on small overlapping areas between different images to make judgments when processing changes in image perspective, it is crucial to retain the information of these distinguishing areas. In addition, due to the difference in training objectives, that is, the general representation ability of pre-trained VFM often pays more attention to foreground objects, while the VPR task relies more on background areas such as buildings. The present application inserts a dynamic power normalization (DPN) layer into the backbone network to enhance the retention of information in these key background areas. In the case of a frozen backbone network, the method provided by the present application can achieve excellent performance only through training with adapters and centroid-free probing. The specific embodiment of the present invention provides an image position recognition method for a basic visual model, Figure 1 and Figure 2 Shown, including:

[0041] (1) The obtained pictures are used as training samples, and the real IDs of the obtained pictures are used as labels. Multiple training samples are used to construct a training sample set.

[0042] (2) Construct a training model, which includes a Patchify module, an improved ViT module and a centroid-free detection module.

[0043] In a specific embodiment of the present invention, the Patchify module is used to divide training samples into multiple local vectors. In one specific embodiment, the Patchify module discretizes the training samples, allowing images to be processed by the Transformer model similarly to text. This specific embodiment of the present invention requires cutting the input image into patches of a fixed size (e.g., 14*14), with a total number of patches being L. Each patch image is output to a projection layer consisting of fully connected layers, which projects each patch image into a vector of length C.

[0044] In a specific embodiment of the present invention, multiple local vectors are input into an improved ViT module to obtain multiple local enhanced description vectors. The improved ViT module includes N fine-tuning sub-modules (blocks). Each local vector is passed through the N fine-tuning sub-modules to obtain a corresponding local enhanced description vector. The Fea provided in a specific embodiment of the present invention consists of L local enhanced description vectors of length C. It is called a local enhanced description vector because the receptive field represented by each vector only occupies a small area of ​​the corresponding image.

[0045] Each fine-tuning submodule includes a first normalization layer (LayersNorm), a multi-head attention layer, a second normalization layer (LayersNorm), a fully connected layer (Multilayer Perceptron, MLP), and a first dynamic power normalization layer (DPN). Each local vector is sequentially input into the first linear layer and the multi-head attention layer to obtain the attention feature. The attention feature and the local vector are first summed to obtain the first sum feature. The first sum feature is sequentially passed through the second linear layer, the fully connected layer, and the first dynamic power normalization layer to obtain the enhanced feature. The enhanced feature and the first sum feature are then summed again to obtain the second sum feature.

[0046] The first dynamic power normalization layer provided in the specific embodiment of the present invention is connected to the fully connected layer, and is used to adjust the feature vector output by the fully connected layer based on the L2 norm constructed by the dynamic power to enhance the background information characteristics. The dynamic power is obtained by linearly transforming the features output by the fully connected layer after average pooling. The dynamic power is a scalar. The dynamic power provided in the specific embodiment of the present invention is obtained based on the feature prediction of the fully connected layer output. Therefore, it can reflect the richness of the background information contained in the feature. If the task-related information in the picture is scarce, the corresponding features are weakened or even eliminated based on the dynamic power. If the information is rich, the corresponding features are enhanced based on the dynamic power.

[0047] In a specific embodiment of the present invention, each element in the feature vector output by the fully connected layer is divided by the L2 norm constructed by the dynamic power to adjust the feature vector output by the fully connected layer. The L2 norm Y constructed by the dynamic power is:

[0048] Y=||sign(X)|X| a ||2

[0049] Where sign(﹒) is the sign function, X is the feature vector output by the fully connected layer, α is the dynamic power, and ||·||2 is the L2 norm.

[0050] The dynamic power α provided in the specific embodiment of the present invention is:

[0051] α=Sigmoid(Linear(Relu(Linear(Avg_pool(X)))))

[0052] Among them, X is the feature vector output by the fully connected layer, Linear represents the linear layer, Sigmoid and Relu are two different activation functions, and Avg_pool represents average pooling.

[0053] The centroid-free detection module provided in a specific embodiment of the present invention includes a second dynamic power normalization layer and two linear transformation layers, wherein one linear transformation layer includes a Sigmoid function for eliminating the centroid, and multiple local enhancement description vectors are input into the two linear transformation layers for dimension compression and then fused, and the fusion results are feature enhanced by the second dynamic power normalization layer to obtain a global vector.

[0054] The fusion result G provided by the specific embodiment of the present invention is:

[0055]

[0056] in, It is the first feature conversion layer, which consists of two linear layers. Multiple local enhancement description vectors Fea are input into the first feature conversion layer to obtain the first compressed vector. The size of the first compressed vector is L*D, where D is a value smaller than C. Therefore, the latitude of the feature can be compressed to save feature storage space. The second feature conversion layer has the same architecture as the first, but it uses multiple local enhancement description vectors (Fea) as inputs to the second feature conversion layer. The resulting compressed vector is of size L*K. This compressed vector is then sigmoided column-by-column through a sigmoid function, ensuring that the sum of the weights assigned to each implicit centroid is always 1. This prevents any implicit centroid from having too much or too little influence on training. This reduces the number of parameters, speeds up training, and avoids the influence of centroids on model training.

[0057] In one embodiment, the first compressed vector for:

[0058]

[0059] Among them, Fea is a number of local enhancement description vectors, Linear represents the linear layer, and Relu is the activation function.

[0060] The specific embodiment of the present invention again uses the DPN layer to perform feature enhancement to obtain the global vector The global vector for:

[0061]

[0062] b=Sigmoid(Linear(Relu(Linear(Avg_pool(G))))).

[0063] (3) A loss function is constructed based on the predicted ID and label corresponding to the global vector. The first dynamic power normalization layer and the centroid-free detection module are trained using the loss function through the training sample set to obtain a basic visual model. The present invention only trains the first dynamic power normalization layer and the centroid-free detection module. Since the number of parameters of these two modules is relatively small, the training is efficient.

[0064] (4) When applied, the image to be identified is input into the basic visual model to obtain a predicted global vector, and the predicted global vector is matched with the global vector of the image in the database. The ID of the image with the best distance match is used as the position of the image to be identified.

[0065] like Figure 3 As shown, a specific embodiment of the present invention also provides an image position recognition device for a basic visual model, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the image position recognition method of the basic visual model.

Claims

1. A method for image position recognition based on a basic visual model, characterized in that: include: The obtained pictures are used as training samples, and the real IDs of the obtained pictures are used as labels. A training sample set is constructed by multiple training samples. Constructing a training model, the training model includes an improved ViT module and a centroid-free detection module, dividing the training sample into multiple image blocks, inputting the multiple image blocks into the improved ViT module to obtain multiple local enhanced description vectors, the improved ViT module includes a first dynamic power normalization layer, the first dynamic power normalization layer is connected to the fully connected layer of the ViT module, and is used to adjust the feature vector output by the fully connected layer based on the L2 norm constructed by the dynamic power to enhance the background information feature, the dynamic power is obtained by averaging the features output by the fully connected layer and then performing a linear transformation, the no centroid detection module includes a second dynamic power normalization layer and two linear transformation layers, wherein one linear transformation layer includes a Sigmoid function, the multiple local enhanced description vectors are input into the two linear transformation layers for dimensional compression and then fused, and the fusion result is feature enhanced by the second dynamic power normalization layer to obtain a global vector; A loss function is constructed based on the predicted ID and label corresponding to the global vector. The first dynamic power normalization layer and the centroid-free detection module are trained using the loss function through the training sample set to obtain the basic visual model. When applied, the image to be identified is input into the basic visual model to obtain a predicted global vector, and the predicted global vector is matched with the global vector of the image in the database to obtain the position of the image to be identified; Each element of the feature vector output by the fully connected layer is divided by the L2 norm constructed by the dynamic power to adjust the feature vector output by the fully connected layer. The L2 norm Y constructed by the dynamic power is: Y=||sign(X)|X| a ||2 Among them, sign(﹒) is the sign function, X is the feature vector output by the fully connected layer, α is the dynamic power, and ‖·‖2 is the L2 norm.

2. The image position recognition method of the basic visual model according to claim 1, characterized in that: Multiple local enhancement description vectors are input into two linear transformation layers for dimension compression and fusion, including: The two linear transformation layers include a first linear transformation layer and a second linear transformation layer; The first linear transformation layer is a first feature transformation layer composed of two linear layers, and the dimensions of the multiple local enhancement description vectors are compressed by the first feature transformation layer to obtain a first compressed vector; The second linear transformation layer includes a second feature conversion layer composed of two linear layers and a Sigmoid function, and the multiple local enhancement description vectors are sequentially input into the second feature conversion layer and the Sigmoid function to obtain a second compressed vector; The first compressed vector is multiplied by the second compressed vector to complete vector fusion.

3. The image position recognition method of the basic visual model according to claim 2, characterized in that: The dimensions of the multiple local enhancement description vectors are compressed by the first feature conversion layer to obtain a first compressed vector. The first compressed vector for: Among them, Fea is a number of local enhancement description vectors, Linear(﹒) represents the linear layer, and Relu(﹒) is the activation function.

4. The image position recognition method of the basic visual model according to claim 1, characterized in that: The dynamic power is obtained by performing linear transformation after average pooling of the features output by the fully connected layer. The dynamic power α is: α=Sigmoid(Linear(Relu(Linear(Avg_pool(X))))) Where X is the feature vector output by the fully connected layer, Linear(﹒) represents the linear layer, Relu(﹒) is the activation function, and Avg_pool(﹒) represents average pooling.

5. The image position recognition method of the basic visual model according to claim 1 is characterized in that: The improved ViT module includes multiple fine-tuning submodules, and each local vector is passed through the multiple fine-tuning submodules to obtain a corresponding local enhanced description vector; Each fine-tuning submodule consists of a first normalization layer, a multi-head attention layer, a second normalization layer, a fully connected layer, and a first dynamic power normalization layer; Among them, each local vector is input into the first normalization layer and the multi-head attention layer in turn to obtain the attention feature, the attention feature and the local vector are summed for the first time to obtain the first sum feature, the first sum feature is passed through the second normalization layer, the fully connected layer and the first dynamic power normalization layer in turn to obtain the enhanced feature, and the enhanced feature and the first sum feature are summed for the second time to obtain the second sum feature.

6. The image position recognition method based on the basic visual model according to claim 1, characterized in that: The training samples are divided into multiple image blocks, including: The training samples are cut into block images of fixed size, and each block image is input into a projection layer composed of a fully connected layer, so that each block image is projected into a corresponding local vector.

7. An image position recognition device based on a basic visual model, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the method for image position recognition of a basic visual model according to any one of claims 1 to 6 is implemented.

Images