Visual place recognition method, device and equipment enhanced by cross-image associated information

Abstract

The application relates to a visual location recognition method, device and equipment based on cross-image associated information enhancement. A trained global descriptor extraction network based on cross-image associated information enhancement is used to extract a database global descriptor of position image data in a map database and a query global descriptor of visual image data to be subjected to location recognition. The network comprises a multi-layer feature mixing unit using a large model to perform multi-layer feature extraction and fuse the multi-layer features in a channel and a spatial dimension to obtain multi-layer mixed features; a multi-level region feature aggregation unit performing multi-level region feature aggregation to obtain a region feature set; an image feature self-enhancement coding unit performing self-enhancement on the region feature set to obtain a global descriptor; and a location position in the visual image data is recognized according to the similarity between the query global descriptor and the database global descriptor, so that stable and high-accuracy positioning of a visual location recognition task is realized.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, apparatus, and device for visual location recognition with enhanced cross-image correlation information. Background Art

[0002] Visual place recognition is the task of estimating the geographic location of a query image based on its visual correlation with a database of images. Visual place recognition is a key technology for autonomous robot positioning and navigation, with important applications in areas such as autonomous driving and augmented reality. Over the past decade, deep learning methods have been widely applied to visual place recognition.

[0003] In the existing technology, there is a method that uses the large model DINOv2 as the backbone network and only uses the features output by its last layer as local features. The local features are made more suitable for visual place recognition tasks through fine-tuning methods, and the local features are aggregated in the aggregation layer to obtain global features. Among them, the CricaVPR method adds an additional cross-image information exchange encoder after the aggregation layer. The input image features enter the encoder in a sequence, and the output image features are optimized by communication with other image features in the sequence. By fine-tuning the large model, local features that are more suitable for place recognition tasks are obtained, but at the same time, the rich feature information contained in the original network is lost. The output features of the cross-image information exchange encoder proposed by CricaVPR change with the number of input images. The output features of the same image in different image sequences are also different, making it difficult to put into practical use. Summary of the Invention

[0004] Based on this, it is necessary to provide a visual location recognition method, device and equipment that can achieve stable and high-accuracy positioning and enhance cross-image correlation information to address the above technical problems.

[0005] A visual place recognition method with enhanced cross-image correlation information, the method comprising:

[0006] Acquiring visual image data for location recognition;

[0007] A query global descriptor of the visual image data is extracted using a trained global descriptor extraction network enhanced with cross-image correlation information, wherein the trained global descriptor extraction network enhanced with cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit. The multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature. The multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set. The data in the regional feature set is rearranged and input into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature, thereby obtaining a stable and discriminative query global descriptor.

[0008] Extracting a database global descriptor having position image data in a map database using the trained global descriptor extraction network enhanced with cross-image correlation information;

[0009] Locations in the visual image data are identified based on similarities between the query global descriptor and database global descriptors.

[0010] In one embodiment, the visual image data is one visual image or multiple visual images. When the visual image data is multiple visual images, the trained global descriptor extraction network with enhanced cross-image correlation information simultaneously extracts the query global descriptor of each visual image.

[0011] In one embodiment, in the multi-level feature mixing unit:

[0012] After dividing the visual image input to the multi-level feature mixing unit into a plurality of image blocks, one-dimensionally flattening and linearly projecting the image blocks are performed to obtain a sequence of block vectors;

[0013] Prepending a learnable vector to each block vector in the block vector sequence, and then superimposing an additional position code on each vector to obtain a vector sequence;

[0014] Inputting the vector sequence into the DINOv2 large model to extract features and obtain block features;

[0015] The block features are processed in sequence using a convolutional layer, a ReLU activation function, and a block element mixer to obtain the multi-level mixed features.

[0016] In one embodiment, in the multi-level regional feature aggregation unit:

[0017] Rearranging the multi-level mixed features into a two-dimensional feature map;

[0018] Based on the two-dimensional feature map, GeM pooling operations are performed on 14 regions at three different levels: 1×1, 2×2, and 3×3, to generate regional features of multiple dimensions;

[0019] After the regional features of the multiple dimensions are connected in series, the regional feature set is obtained.

[0020] In one embodiment, the image feature self-enhancement encoding unit is composed of a basic Transformer network structure.

[0021] In one embodiment, when training the global descriptor extraction network for cross-image correlation information enhancement, two training stages are included:

[0022] In the first training stage, the teacher network is first trained to obtain a trained teacher network, wherein the teacher network is a CricaVPR network optimized by the multi-level feature hybrid unit;

[0023] In the second training stage, the trained teacher network is used to supervise the training of the global descriptor extraction network enhanced with cross-image correlation information, thereby obtaining the trained global descriptor extraction network enhanced with cross-image correlation information.

[0024] In one embodiment, when training the teacher network and the global descriptor extraction network enhanced with cross-image correlation information, a small batch training method is adopted, and each batch of training data includes a plurality of sample images corresponding to different locations, wherein a preset number of sample images are set for each location;

[0025] The teacher network and the global descriptor extraction network enhanced with cross-image correlation information are trained once using a batch of training data.

[0026] In one embodiment, in the second stage:

[0027] After preprocessing each sample image in a batch of training data, they are respectively input into the trained teacher network and the global descriptor extraction network enhanced with cross-image correlation information, and the corresponding teacher prediction results and student prediction results are obtained;

[0028] Calculating a mean square error loss function based on the teacher's prediction results and the student's prediction results, and calculating a multi-similarity loss function based on the student's prediction results;

[0029] According to the two calculation results, the learnable parameters in the global descriptor extraction network for cross-image correlation information enhancement are adjusted until a specified number of iterations is reached to obtain the trained global descriptor extraction network for cross-image correlation information enhancement.

[0030] The present application also provides a visual place recognition device with enhanced cross-image correlation information, the device comprising:

[0031] The module for acquiring images of locations to be identified is used to acquire visual image data of locations to be identified;

[0032] A query global descriptor acquisition module is used to extract the query global descriptor of the visual image data using a trained global descriptor extraction network enhanced with cross-image correlation information, wherein the trained global descriptor extraction network enhanced with cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit. The multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain multi-level mixed features. The multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed features to obtain a regional feature set. The data in the regional feature set is rearranged and input into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature to obtain a stable and discriminative query global descriptor.

[0033] A database global descriptor acquisition module is used to extract database global descriptors with location image data in a map database using the trained global descriptor extraction network enhanced with cross-image correlation information;

[0034] The location recognition module is used to recognize the location of the visual image data based on the similarity between the query global descriptor and the database global descriptor.

[0035] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0036] Acquiring visual image data for location recognition;

[0037] A query global descriptor of the visual image data is extracted using a trained global descriptor extraction network enhanced with cross-image correlation information, wherein the trained global descriptor extraction network enhanced with cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit. The multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature. The multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set. The data in the regional feature set is rearranged and input into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature, thereby obtaining a stable and discriminative query global descriptor.

[0038] Extracting a database global descriptor having position image data in a map database using the trained global descriptor extraction network enhanced with cross-image correlation information;

[0039] Locations in the visual image data are identified based on similarities between the query global descriptor and database global descriptors.

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0041] Acquiring visual image data for location recognition;

[0042] A query global descriptor of the visual image data is extracted using a trained global descriptor extraction network enhanced with cross-image correlation information, wherein the trained global descriptor extraction network enhanced with cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit. The multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature. The multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set. The data in the regional feature set is rearranged and input into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature, thereby obtaining a stable and discriminative query global descriptor.

[0043] Extracting a database global descriptor having position image data in a map database using the trained global descriptor extraction network enhanced with cross-image correlation information;

[0044] identify the location position in the visual image data according to the similarity between the query global descriptor and the database global descriptor.

[0045] The visual location recognition method, device and equipment with cross-image correlation information enhancement described above, by using the trained cross-image correlation information enhanced global descriptor extraction network, extracts the query global descriptor of the visual image data, wherein the cross-image correlation information enhanced global descriptor extraction network includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit and an image feature self-enhancement coding unit. The multi-level feature mixing unit uses the DINOv2 large model to extract multi-layer features from the visual image data, and fuses the extracted multi-layer features in the channel dimension and the spatial dimension to obtain multi-level mixed features. The multi-level regional feature aggregation unit aggregates the multi-level mixed features to obtain a regional feature set. The data in the regional feature set is rearranged and input to the image feature self-enhancement coding unit for self-enhancement of each regional feature, to obtain the query global descriptor with distinguishability. At the same time, the trained cross-image correlation information enhanced global descriptor extraction network is used to extract the database global descriptor of the position image data in the map database. Finally, the location position in the visual image data is identified according to the similarity between the query global descriptor and the database global descriptor. The method can realize stable and high-accuracy positioning of the visual location recognition task. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 A flowchart of the cross-image correlation information enhanced visual location recognition method in one embodiment;

[0047] Figure 2 A general step and test flowchart of the visual location recognition task in one embodiment;

[0048] Figure 3 A structure diagram of the cross-image correlation information enhanced global descriptor extraction network in one embodiment;

[0049] Figure 4 A structure diagram of the multi-level feature mixing unit in one embodiment;

[0050] Figure 5 A structure diagram of the multi-level regional feature aggregation unit in one embodiment;

[0051] Figure 6 A network structure diagram of the teacher network Super-CricaVPR in one embodiment;

[0052] Figure 7 A training flowchart of the teacher network in one embodiment;

[0053] Figure 8 A flowchart of training a global descriptor extraction network for enhanced cross-image correlation information in one embodiment;

[0054] Figure 9 A schematic diagram showing the comparison of qualitative results of this method and other visual place recognition methods in an experiment;

[0055] Figure 10 A structural block diagram of a visual place recognition device with enhanced cross-image correlation information according to one embodiment;

[0056] Figure 11 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0058] To address the problems existing in the existing visual image place recognition method developed based on the large model DINOv2, as shown in Figure 1, a visual place recognition method with enhanced cross-image correlation information is provided, which specifically includes the following steps:

[0059] Step S100: Acquire visual image data for location recognition.

[0060] Step S110, using the trained global descriptor extraction network for cross-image correlation information enhancement, extract the query global descriptor of the visual image data, wherein the trained global descriptor extraction network for cross-image correlation information enhancement includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit and an image feature self-enhancement encoding unit, the multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain multi-level mixed features, the multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed features to obtain a regional feature set, and rearranges the data in the regional feature set and inputs it into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature to obtain a stable and discriminative query global descriptor.

[0061] Step S120 , using the trained global descriptor extraction network enhanced with cross-image correlation information, extracting database global descriptors with location image data in the map database.

[0062] Step S130 , identifying a location in the visual image data based on the similarity between the query global descriptor and the database global descriptor.

[0063] In this embodiment, based on the conventional visual place recognition task framework, such as Figure 2 As shown in the figure, by fully utilizing the rich visual information contained in the large model DINOv2 and the feature optimization brought by the cross-image information exchange encoder in the CricaVPR network, a global descriptor extraction network is proposed, which includes a multi-level feature mixing unit and a self-enhanced encoder with implicit image association information (i.e., image feature self-enhanced coding unit), namely a global descriptor extraction network enhanced with cross-image association information. In this network, the multi-level feature mixing unit utilizes the multi-layer output features of DINOv2 and fully fuses the features in the channel dimension and spatial dimension respectively. In addition, the self-enhanced encoder distills the image association information explicitly implemented in the cross-image information exchange encoder into its own weight model, so that the output global features can be stably enhanced by the image association information.

[0064] Compared to existing visual place recognition methods, the method proposed in this application can fully utilize the rich visual information contained in the large DINOv2 model to provide local features relevant to the visual place recognition task. Compared to the advanced method CricaVPR for visual place recognition tasks, the global descriptor extraction network in this method can output global descriptors that are stably enhanced by image correlation information, achieving stable and high-accuracy positioning for visual place recognition tasks.

[0065] In step S100, the obtained visual image data to be used for location recognition can be one visual image or multiple visual images. When the visual image data is multiple visual images, the trained global descriptor extraction network with enhanced cross-image correlation information simultaneously extracts the query global descriptor of each visual image.

[0066] In this embodiment, the visual image is an RGB image.

[0067] like Figure 3 As shown in the figure, the data processing process of the global descriptor extraction network with cross-image correlation information enhancement is shown, wherein the network includes three processing units in sequence, namely a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit.

[0068] In this embodiment, in the multi-level feature mixing unit: the visual image input to the multi-level feature mixing unit is divided into multiple image blocks, which are flattened in one dimension and linearly projected to obtain a block vector sequence, a learnable vector is prepended to each block vector in the block vector sequence, and an additional position code is superimposed on each vector to obtain a vector sequence, and then the vector sequence is input into the DINOv2 large model for feature extraction to obtain block features, and then the block features are processed in sequence using convolutional layers, ReLU activation functions and block element mixers to obtain multi-level mixed features.

[0069] Specifically, such as Figure 4 As shown, taking a single input image as an example, the two-dimensional input image x∈R H×W×C (length H, width W, C is the original number of channels), is first divided into N image blocks, then flattened in one dimension, and then converted into block vectors through linear projection In addition, a learnable [class] vector is prepended to the block vector to form the vector To save the position information, additional position encoding is superimposed on the vector x0 The resulting vector It can be input into the subsequent DINOv2 model for feature extraction.

[0070] The large model DINOv2 consists of Transformer layers, each of which contains a multi-head self-attention module (MSA) and a multi-layer perceptron (MLP). There is a layer normalization operation before these two modules and a residual connection operation after these two modules.

[0071] The formulas involved in the above operations are as follows:

[0072] z0=x0+E pos

[0073] z l ′=MSA(LN(z l-1 ))+z l-1

[0074] z l =MLP(LN(z l ′))+z l '

[0075] In the above formula, l=1,...,L.

[0076] Since the DINOv2 model is trained on a large-scale unsupervised dataset and contains rich visual feature information, the global descriptor extraction network for enhancing cross-image correlation information directly uses the frozen-weight DINOv2 model as the backbone network, and the network weights of DINOv2 are not updated during subsequent model training.

[0077] Feature z output by DINOv2 i Include class features and block features Both can be used to describe the global information of the image. To reduce the amount of computation, the class features are discarded in this method and only the output block features are considered. In order to make full use of the features output by DINOv2, the block features output by the last M layers are concatenated in the global descriptor extraction network with enhanced cross-image correlation information. get First, a 1×1 convolution is used to fully integrate the channel information of the feature and reduce the channel dimension. Then, the ReLU activation function is used to enhance the nonlinearity of the feature and flatten the block feature for subsequent operations.

[0078] The formulas involved in the above operations are as follows:

[0079]

[0080] Y′=Concat(y L-M ;…;y L )

[0081] Y = Flatten(ReLU(Conv(Y′)))

[0082] In the above formula, i=LM,...,L and

[0083] Furthermore, a block mixer is used to fully communicate and fuse the spatial information between N different blocks. The multilayer perceptron in the block mixer operates independently on each channel, and its parameters are shared across all channels. The multilayer perceptron consists of two fully connected layers and a ReLU function, and the number of feature channels in the output is the same as the number of feature channels in the input. The formula involved in this process is as follows:

[0084] Y={Y j}

[0085] F j =Y j +W2σ(W1(LN(Y j ))

[0086] F={F j}

[0087] In the above formula, j = 1, ..., C2, σ represents the ReLU activation function, W2∈R N×P , W3∈R P×N And F is the output multi-level mixed feature.

[0088] In this embodiment, in the multi-level regional feature aggregation unit: the multi-level mixed features are rearranged into a two-dimensional feature map. Based on the two-dimensional feature map, GeM pooling operations are performed in 14 regions at three different levels of 1×1, 2×2, and 3×3 to generate regional features of multiple dimensions. Finally, the regional features of multiple dimensions are connected in series to obtain a regional feature set, such as Figure 5 shown.

[0089] Specifically, multi-level hybrid features aggregate multi-level regional features in the aggregation layer, such as Figure 5 The multi-level mixed features are first rearranged into a two-dimensional feature map (length H, width W, number of channels C2), and then divided into 14 regions of three different levels of 1×1, 2×2, and 3×3 in the aggregation layer for GeM pooling operations to generate 14 regional features of dimension C2. These features are concatenated together to obtain the output of the aggregation layer.

[0090] In this embodiment, the image feature self-enhancement encoding unit is composed of a basic structure of a Transformer network. At this time, taking the input visual image data as B visual images as an example, the regional feature set The rearrangement is to let each regional feature enter the self-enhancing encoder one by one. The sample batch input to the encoder becomes 14B and the sequence size becomes 1. The rearranged regional feature set is The self-enhanced encoder consists of a basic Transformer structure, in which 14B regional features can only be self-enhanced encoded in the multi-head self-attention module. The regional features after self-enhanced encoding can be rearranged and L2 normalized to form a stable and discriminative global descriptor.

[0091] In this embodiment, a method for training the global descriptor extraction network for cross-image correlation information enhancement is also provided. The method includes two training phases. In the first training phase, a teacher network is trained to obtain a trained teacher network, wherein the teacher network is a CricaVPR network optimized using a multi-level feature hybrid unit. In the second training phase, the trained teacher network is used to supervise the training of the global descriptor extraction network for cross-image correlation information enhancement, thereby obtaining a trained global descriptor extraction network for cross-image correlation information enhancement.

[0092] Specifically, the SciceVPR (cross-image correlation information enhanced global descriptor extraction network) training process proposed in this paper is divided into two stages. Taking a batch of B RGB images as an example, the training process is briefly described. First, the Super-CricaVPR model structure is obtained by optimizing the CricaVPR model structure using the multi-level feature mixing module as shown in Figure 6 . The small batch training method is used for separate training to obtain the Super-CricaVPR network model that can explicitly use cross-image communication information. The training process of this stage is shown in Figure 7 . Then, the SciceVPR network using the multi-level feature mixing unit and the self-enhanced encoder is initialized. The trained Super-CricaVPR is used as a teacher network to supervise the small batch training process of the student network SciceVPR, and the SciceVPR network model that implicitly contains image correlation information is obtained. The training process of this stage is shown in Figure 8 .

[0093] Further, in order to enable the cross-image correlation information enhanced global descriptor extraction network to learn the feature optimization brought by the cross-image information communication encoder in Super-CricaVPR, in this embodiment, the cross-image correlation information enhanced global descriptor extraction network is trained in the form of Super-CricaVPR as a teacher network. At the same time, Super-CricaVPR replaces the fine-tuned large model DINOv2 with the multi-level feature mixing unit based on the original architecture of CricaVPR.

[0094] Further, when training the teacher network and the cross-image correlation information enhanced global descriptor extraction network, the small batch training method is used. In each batch of training data, multiple sample images corresponding to different locations are included. For each location, a predetermined number of sample images are set. A batch of training data is used to train the teacher network and the cross-image correlation information enhanced global descriptor extraction network once.

[0095] In this embodiment, when training the teacher network and the cross-image correlation information enhanced global descriptor extraction network using training data, the training data is preprocessed. The resolution of the sample images in the training data is uniformly scaled to 224x224. Then, three random data augmentation methods provided in the torchvision.transforms.RandAugment function are used. If interpolation operation is needed, bilinear interpolation is used. Finally, the images are normalized using the mean and standard deviation of the ImageNet dataset.

[0096] Furthermore, in the teacher network, the first two units, the multi-level feature mixing unit and the multi-level regional feature aggregation unit, have been explained in the previous article and will not be repeated here. In the cross-image information exchange encoder, the regional feature set output by the multi-level regional feature aggregation unit is encoded for information exchange. This stage is introduced by taking the number of batch samples as B as an example. The most critical step in realizing cross-image information exchange encoding is to encode the regional feature set. The rearrangement is to connect all x-th regional features in a batch B in series and input them into the cross-image information exchange encoder in a sequence. At this time, the sample batch input to the encoder becomes 14, the sequence size becomes B, and the rearranged regional feature set is

[0097] Specifically, the cross-image information exchange encoder consists of two basic Transformer structures. In this encoder, all regional features in sequence B can be exchanged with each other in the multi-head self-attention module. The regional features after cross-image information exchange can be rearranged and L2 normalized to form a discriminative global descriptor.

[0098] Furthermore, in the first training phase, the Super-CricaVPR model (teacher model) was trained using a mini-batch training method. The batch size was set to 288, including images from 72 different locations (4 images per location). The multi-similarity loss function and the online difficult sample mining strategy were used to find B1 positive sample pairs (two sample images of the same location, but with low global descriptor similarity) and B2 negative sample pairs (two sample images of different locations, but with high global descriptor similarity) that were difficult to distinguish among the 288 images. The multi-similarity loss function was then calculated, and the network parameters were then updated through backpropagation. Training was stopped after reaching the specified number of iterations, otherwise training continued. The calculation of the multi-similarity loss function is expressed as:

[0099]

[0100] In formula (1), B is the batch size 288, for each query image I q , is the set of positive sample index p, S is the set of negative sample index n. pq and S qn They are positive sample pairs {I q ,I p} and negative sample pairs {I q ,I n}, α, β and λ are hyper parameters.

[0101] In this embodiment, in the second stage: after preprocessing each sample image in a batch of training data, the trained teacher network and the global descriptor extraction network with cross-image correlation information enhancement are respectively input, and the corresponding teacher prediction results and student prediction results are obtained, the mean square error loss function is calculated based on the teacher prediction results and the student prediction results, and the multi-similarity loss function is calculated based on the student prediction results. According to the two calculation results, the learnable parameters in the global descriptor extraction network with cross-image correlation information enhancement are adjusted until convergence, and a trained global descriptor extraction network with cross-image correlation information enhancement is obtained.

[0102] Specifically, the input image first undergoes the same image preprocessing as the training Super-CricaVPR, and then enters the trained Super-CricaVPR network model and the SciceVPR network to be trained. The multi-level hybrid feature extraction and multi-level regional feature aggregation stages experienced in the SciceVPR network model are exactly the same as those in Super-CricaVPR. The difference lies in the image feature self-enhancement encoding stage in SciceVPR. Figure 3 The operations involved after the aggregation layer and the image feature self-enhancement coding unit have been explained above and will not be explained here.

[0103] Furthermore, the training of the SciceVPR network model also adopts a small batch training method, with a batch size of 288, including images from 72 different locations (4 images per location), and also adopts a multi-similarity loss function and an online difficult sample mining strategy. The difference is that the supervision signal of the Super-CricaVPR teacher network is added in the loss calculation process. The global descriptor generated by SciceVPR is made as similar as possible to the global descriptor generated by Super-CricaVPR, so that the weights of the self-enhanced encoder in SciceVPR contain cross-image correlation information, and the mean square error function shown in formula (2) is used for loss calculation. The mean square error function is expressed as:

[0104]

[0105] Then, according to formula (1) and formula (2), the total loss function for training the SciceVPR network is obtained, which is expressed as:

[0106] L 总 =γL1+ηL2 (3)

[0107] In formula (3), γ and η are hyperparameters.

[0108] The SciceVPR network parameters are then updated through backpropagation. Training stops when the specified number of iterations is reached, otherwise training continues. During this process, Super-CricaVPR uses the trained network parameters and keeps them fixed throughout the SciceVPR training phase without parameter updates.

[0109] In steps S120 and S130, the data with specific location information in the database is input into the trained global descriptor extraction network enhanced with cross-image correlation information to obtain the database global descriptor. The location in the visual image is then determined by calculating the distance between the query global descriptor and the database global descriptor.

[0110] To evaluate the performance of the global descriptor extraction network with enhanced cross-image correlation information proposed in this paper, the model was trained on the GSV-Cities dataset, a public dataset widely used in visual place recognition tasks, and tested on other public datasets: Pitts30k-test, Tokyo24 / 7, MSLS-val / challenge, AmsterdamTime, and SVOX datasets. The main features of these datasets are shown in Table 1.

[0111] Table 1 Overview of the visual place recognition test dataset used in the experiment

[0112]

[0113] The Pitts30k dataset contains 10,000 database images downloaded from Google Street View, with GPS tags for each train / val / test dataset. The model is evaluated on the test dataset, which contains 6,816 query images that are taken at different times and years than the database images.

[0114] The Tokyo24 / 7 dataset consists of 75,984 database images from Google Street View and 315 query images taken with mobile phones on sidewalks, both of which are GPS-tagged. The database images are all daytime images, while the query images can be either daytime or nighttime.

[0115] MSLS is a large-scale dataset for visual place recognition in urban and suburban areas, containing image sequences spanning nine years. MSLS addresses the challenges of visual place recognition, including seasonal and weather changes, illumination variations at different times of day, and varying viewpoints. Images in MSLS are GPS-tagged. Models are tested on a public validation set (MSLS-val) and a test set (MSLS-challenge) with metadata preserved.

[0116] The AmsterTime dataset contains 1,231 pairs of images of urban areas. Each pair consists of a historical grayscale query image and a corresponding modern RGB reference image representing the same location. This dataset is challenging due to domain variations in viewpoint, modality (RGB vs. grayscale), and very long time spans.

[0117] SVOX is a cross-domain dataset for evaluating visual place recognition models in various weather conditions. The dataset covers the city of Oxford, with dataset images from Google Street View and query images from the Oxford RobotCar dataset, which covers a variety of weather and lighting conditions, such as cloudy, rainy, sunny, snowy, and night.

[0118] Test process: First, the database image and query image are preprocessed (in the test phase, only the images need to be uniformly scaled to 224×224 and normalized using the mean and standard deviation of the ImageNet dataset). The database image is passed through the SciceVPR network model to generate a database global descriptor and stored. The query image is passed through the SciceVPR network model to generate a query global descriptor. The nearest neighbor search method is used to find the database image that is most similar to the query image, and the geographical location of the database image is used as the query image location. Figure 2 shown.

[0119] Evaluation Criteria: Recall@N is a metric for visual place recognition tasks, defined as the percentage of query images for which at least one of the top N predictions is from the same location as the query image, abbreviated as R@N. For the GPS-tagged Pitts30k, Tokyo24 / 7, MSLS, and SVOX datasets, predicted database images are considered to be from the same location if they are within 25 meters of the query image. AmsterTime is a collection of image pairs where only the database image corresponding to the query image is from the same location as the query image.

[0120] Experimental implementation details: Super-CricaVPR and SciceVPR were trained on two NVIDIA GeForce RTX 3090s, with a batch size of 288 (images from 72 different locations in GSV-Cities, each containing 4 images). The DINOv2 backbone network used was the ViT-B / 14 model. The output features of the last four layers of DINOv2 were concatenated, and the number of feature channels after concatenation was reduced to 768 through 1×1 convolution. The number of block mixers was 2, and the hidden dimension was 16×16. Finally, the global descriptor dimensions of Super-CricaVPR and SciceVPR were both 14×768, and the global descriptor output by SciceVPR was further reduced to 4096 through PCA. The hyperparameters in formula (1) are α=1, β=50, λ=0; the hyperparameters in formula (3) are γ=1, η=1. The model was trained using the Adam optimizer, with an initial learning rate of 0.0001 and multiplied by 0.5 after every three training rounds. The number of training rounds for Super-CricaVPR was set to 10, and for SciceVPR was set to 2.

[0121] Experimental Comparison: This paper compares the results of baseline methods and our method on a variety of challenging visual place recognition datasets from the past three years. Because the CricaVPR model's test results are significantly affected by the dataset arrangement and the batch size of the input images, for the sake of fairness, the CricaVPR model was tested with a batch size of 1, named CricaVPR-single. The comparison results are shown in Tables 2 and 3. In Tables 2 and 3, R@1, R@5, and R@10 are the accuracy evaluation criteria. The best results are indicated in bold, and the second-best results are underlined.

[0122] Table 2 Comparison results of this method with other mainstream state-of-the-art visual place recognition models on various benchmark datasets

[0123]

[0124] Table 3 Comparison results of this method with other mainstream state-of-the-art visual place recognition models on various challenging test datasets

[0125]

[0126] As shown in Table 2, SciceVPR surpasses all other models in terms of Recall@1 and Recall@10 accuracy. In particular, on the Tokyo24 / 7 dataset, Recall@1 reaches a high of 94.9%. Tokyo24 / 7 includes significant day / night and viewpoint variations, and this result demonstrates the robustness of the SciceVPR model to these changes. Furthermore, on the MSLS-val / challenge dataset, which has a long time span and includes some suburban scenes, SciceVPR also demonstrates good performance, demonstrating the effectiveness of our approach.

[0127] Table 3 further demonstrates the generalization of the SciceVPR model. On the AmsterTime dataset, which exhibits modal and long-term variations, and on the SVOX dataset, which exhibits various weather conditions, the SciceVPR model achieves nearly state-of-the-art place recognition results. On the AmsterTime, SVOX-Night, and SVOX-Sun datasets, the Recall@1 score surpasses the second-place SciceVPR-single by 9.4%, 11.5%, and 6.6%, respectively. In summary, SciceVPR demonstrates excellent performance across a wide range of visual place recognition scenarios.

[0128] Figure 9 The positioning effect of SciceVPR is qualitatively demonstrated, where the green box represents the correct location recognition result and the red box represents the incorrect location recognition result. Figure 9 We demonstrated several challenging localization scenarios for visual place recognition tasks, including large-angle viewpoint changes, lighting variations between day and night, weather changes throughout the year, occlusions caused by buildings, and structural changes to the same location over time. The results showed that SciceVPR can still correctly identify locations in these challenging scenarios, while other methods can only identify similar images in different locations.

[0129] The aforementioned visual place recognition method enhanced with cross-image correlation information exploits the rich visual information contained in the large DINOv2 model and employs a self-enhanced encoder that incorporates implicit image correlation information to provide a stable and discriminative global descriptor, enabling stable and highly accurate localization for visual place recognition tasks. This method preprocesses the input query image and database images. Then, within the SciceVPR network model, it extracts multi-level hybrid features, aggregates multi-level regional features, and self-enhances image features. It outputs a global descriptor implicitly enhanced by image correlation information. A nearest neighbor search is then used to find the database descriptor that is closest to the query global descriptor. The geographic location of this database image is the query image's location. This method significantly improves localization accuracy for visual place recognition tasks.

[0130] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0131] In one embodiment, Figure 10 As shown, a visual place recognition device with enhanced cross-image correlation information is provided, comprising: a place recognition image acquisition module 200, a query global descriptor acquisition module 210, a database global descriptor acquisition module 220, and a place position recognition module 230, wherein:

[0132] The image acquisition module 200 for obtaining the image data for the location to be identified is used to obtain the visual image data for the location to be identified;

[0133] The query global descriptor acquisition module 210 is used to extract the query global descriptor of the visual image data using the trained global descriptor extraction network enhanced by cross-image correlation information, wherein the trained global descriptor extraction network enhanced by cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit and an image feature self-enhancement encoding unit. The multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain multi-level mixed features. The multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed features to obtain a regional feature set. The data in the regional feature set is rearranged and input into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature to obtain a stable and discriminative query global descriptor.

[0134] A database global descriptor acquisition module 220 is configured to extract database global descriptors having location image data in a map database using the trained global descriptor extraction network enhanced with cross-image correlation information;

[0135] The place position identification module 230 is configured to identify the place positions in the visual image data based on the similarity between the query global descriptor and the database global descriptor.

[0136] Regarding the specific limitations of the visual place recognition device with enhanced cross-image correlation information, please refer to the limitations of the visual place recognition method with enhanced cross-image correlation information above, which will not be repeated here. The various modules in the above-mentioned visual place recognition device with enhanced cross-image correlation information can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0137] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a visual place recognition method with enhanced cross-image association information is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0138] Those skilled in the art will understand that Figure 11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0139] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0140] Acquiring visual image data for location recognition;

[0141] A query global descriptor of the visual image data is extracted using a trained global descriptor extraction network enhanced with cross-image correlation information, wherein the trained global descriptor extraction network enhanced with cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit. The multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature. The multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set. The data in the regional feature set is rearranged and input into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature, thereby obtaining a stable and discriminative query global descriptor.

[0142] Extracting a database global descriptor having position image data in a map database using the trained global descriptor extraction network enhanced with cross-image correlation information;

[0143] Locations in the visual image data are identified based on similarities between the query global descriptor and database global descriptors.

[0144] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0145] Acquiring visual image data for location recognition;

[0146] A query global descriptor of the visual image data is extracted using a trained global descriptor extraction network enhanced with cross-image correlation information, wherein the trained global descriptor extraction network enhanced with cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit. The multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature. The multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set. The data in the regional feature set is rearranged and input into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature, thereby obtaining a stable and discriminative query global descriptor.

[0147] Extracting a database global descriptor having position image data in a map database using the trained global descriptor extraction network enhanced with cross-image correlation information;

[0148] Locations in the visual image data are identified based on similarities between the query global descriptor and database global descriptors.

[0149] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0150] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A visual place recognition method with enhanced cross-image correlation information, characterized in that: The method comprises: Acquiring visual image data for location recognition; A query global descriptor of the visual image data is extracted using a trained global descriptor extraction network enhanced with cross-image correlation information, wherein the trained global descriptor extraction network enhanced with cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit. The multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature. The multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set. The data in the regional feature set is rearranged and input into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature, thereby obtaining a stable and discriminative query global descriptor. Extracting a database global descriptor having position image data in a map database using the trained global descriptor extraction network enhanced with cross-image correlation information; Locations in the visual image data are identified based on similarities between the query global descriptor and database global descriptors.

2. The visual place recognition method with enhanced cross-image correlation information according to claim 1, characterized in that: The visual image data is one visual image or multiple visual images. When the visual image data is multiple visual images, the trained global descriptor extraction network enhanced with cross-image correlation information simultaneously extracts query global descriptors of each visual image.

3. The visual place recognition method with enhanced cross-image correlation information according to claim 2, characterized in that: In the multi-level feature mixing unit: After dividing the visual image input to the multi-level feature mixing unit into a plurality of image blocks, one-dimensionally flattening and linearly projecting the image blocks are performed to obtain a sequence of block vectors; Prepending a learnable vector to each block vector in the block vector sequence, and then superimposing an additional position code on each vector to obtain a vector sequence; Inputting the vector sequence into the DINOv2 large model to extract features and obtain block features; The block features are processed in sequence using a convolutional layer, a ReLU activation function, and a block element mixer to obtain the multi-level mixed features.

4. The visual place recognition method with enhanced cross-image correlation information according to claim 2, characterized in that: In the multi-level regional feature aggregation unit: Rearranging the multi-level mixed features into a two-dimensional feature map; Based on the two-dimensional feature map, GeM pooling operations are performed on 14 regions at three different levels: 1×1, 2×2, and 3×3, to generate regional features of multiple dimensions; After the regional features of the multiple dimensions are connected in series, the regional feature set is obtained.

5. The visual place recognition method with enhanced cross-image correlation information according to claim 2, characterized in that: The image feature self-enhancement coding unit is composed of a basic Transformer network structure.

6. The method for visual place recognition with enhanced cross-image correlation information according to any one of claims 1 to 5, characterized in that: When training the global descriptor extraction network for cross-image correlation information enhancement, two training stages are included: In the first training stage, the teacher network is first trained to obtain a trained teacher network, wherein the teacher network is a CricaVPR network optimized by the multi-level feature hybrid unit; In the second training stage, the trained teacher network is used to supervise the training of the global descriptor extraction network enhanced with cross-image correlation information, thereby obtaining the trained global descriptor extraction network enhanced with cross-image correlation information.

7. The visual place recognition method with enhanced cross-image correlation information according to claim 6, characterized in that: When training the teacher network and the global descriptor extraction network enhanced with cross-image correlation information, a small batch training method is adopted. Each batch of training data includes multiple sample images corresponding to different locations, wherein a preset number of sample images are set for each location. The teacher network and the global descriptor extraction network enhanced with cross-image correlation information are trained once using a batch of training data.

8. The visual place recognition method with enhanced cross-image correlation information according to claim 7, characterized in that: In the second training phase: After preprocessing each sample image in a batch of training data, they are respectively input into the trained teacher network and the global descriptor extraction network enhanced with cross-image correlation information, and the corresponding teacher prediction results and student prediction results are obtained; Calculating a mean square error loss function based on the teacher's prediction results and the student's prediction results, and calculating a multi-similarity loss function based on the student's prediction results; According to the two calculation results, the learnable parameters in the global descriptor extraction network for cross-image correlation information enhancement are adjusted until a specified number of iterations is reached to obtain the trained global descriptor extraction network for cross-image correlation information enhancement.

9. A visual location recognition device with enhanced cross-image correlation information, characterized in that: The device comprises: The module for acquiring images of locations to be identified is used to acquire visual image data of locations to be identified; A query global descriptor acquisition module is used to extract the query global descriptor of the visual image data using a trained global descriptor extraction network enhanced with cross-image correlation information, wherein the trained global descriptor extraction network enhanced with cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit. The multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain multi-level mixed features. The multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed features to obtain a regional feature set. The data in the regional feature set is rearranged and input into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature to obtain a stable and discriminative query global descriptor. A database global descriptor acquisition module is used to extract database global descriptors with location image data in a map database using the trained global descriptor extraction network enhanced with cross-image correlation information; The location recognition module is used to recognize the location of the visual image data based on the similarity between the query global descriptor and the database global descriptor.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images