Cross-image associated information enhanced visual location identification method, device and equipment

Abstract

The invention relates to a cross-image associated information enhanced visual location identification method, a cross-image associated information enhanced visual location identification device and cross-image associated information enhanced visual location identification equipment. The method comprises the following steps: respectively extracting a database global descriptor of position image data in a map database and a query global descriptor of visual image data to be subjected to site identification, the multi-level feature mixing unit is used for carrying out multi-level feature fusion on the multi-level features in channel and space dimensions to obtain multi-level mixed features, and the multi-level region feature aggregation unit is used for carrying out multi-level region feature aggregation to obtain a region feature set; and carrying out self-enhancement on the regional feature set to obtain an image feature self-enhancement coding unit of a global descriptor, and identifying a site position in the visual image data according to the similarity between the query global descriptor and the database global descriptor so as to realize stable and high-accuracy positioning of a visual site identification task.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device and apparatus for visual location recognition with enhanced cross-image correlation information. Background Art

[0002] Visual place recognition refers to the task of estimating the geographical location of a query image based on the visual correlation between the query image and database images. Visual place recognition is one of the key technologies for achieving autonomous positioning and navigation of robots, and has important applications in autonomous driving, augmented reality and other fields. In the past decade, deep learning methods have been widely used in visual place recognition tasks.

[0003] In the prior art, there is a method that uses the large model DINOv2 as the backbone network and only uses the features of its last layer output as local features. The local features are made more suitable for visual place recognition tasks through fine-tuning methods, and local features are aggregated in the aggregation layer to obtain global features. Among them, the CricaVPR method adds an additional cross-image information exchange encoder after the aggregation layer. The input image features enter the encoder in a sequence, and the output image features are optimized by communication with other image features in the sequence. By fine-tuning the large model, local features that are more suitable for place recognition tasks are obtained, but at the same time, the rich feature information contained in the original network is lost. The output features of the cross-image information exchange encoder proposed by CricaVPR change with the number of input images. The output features of the same image in different image sequences are also different, which makes it difficult to put into practical use. Summary of the invention

[0004] Based on this, it is necessary to provide a visual location recognition method, device and equipment with enhanced cross-image correlation information that can achieve stable and high-accuracy positioning in response to the above technical problems.

[0005] A visual place recognition method with enhanced cross-image correlation information, the method comprising:

[0006] Acquiring visual image data for location recognition;

[0007] Utilize the trained global descriptor extraction network enhanced with cross-image correlation information to extract the query global descriptor of the visual image data, wherein the trained global descriptor extraction network enhanced with cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit, the multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature, the multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set, rearranges the data in the regional feature set and inputs it into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature, so as to obtain the stable and distinguishable query global descriptor;

[0008] Using the trained global descriptor extraction network enhanced with cross-image correlation information, extracting a database global descriptor having location image data in a map database;

[0009] Locations in the visual image data are identified based on similarities between the query global descriptor and database global descriptors.

[0010] In one embodiment, the visual image data is one visual image or multiple visual images. When the visual image data is multiple visual images, the trained global descriptor extraction network enhanced with cross-image correlation information simultaneously extracts query global descriptors of each visual image.

[0011] In one embodiment, in the multi-level feature mixing unit:

[0012] After dividing the visual image input to the multi-level feature mixing unit into a plurality of image blocks, one-dimensional flattening and linear projection are performed to obtain a block vector sequence;

[0013] Prepending a learnable vector to each block vector in the block vector sequence, and then superimposing an additional position code on each vector to obtain a vector sequence;

[0014] Inputting the vector sequence into the DINOv2 large model to extract features to obtain block features;

[0015] The block features are processed in sequence using a convolutional layer, a ReLU activation function, and a block element mixer to obtain the multi-level mixed features.

[0016] In one embodiment, in the multi-level regional feature aggregation unit:

[0017] Rearranging the multi-level mixed features into a two-dimensional feature map;

[0018] Based on the two-dimensional feature map, GeM pooling operations are performed in 14 regions at three different levels: 1×1, 2×2, and 3×3, to generate regional features of multiple dimensions;

[0019] After the regional features of the multiple dimensions are connected in series, the regional feature set is obtained.

[0020] In one embodiment, the image feature self-enhancement encoding unit is composed of a basic Transformer network structure.

[0021] In one embodiment, when the global descriptor extraction network for cross-image correlation information enhancement is trained, two training stages are included:

[0022] In the first training stage, the teacher network is first trained to obtain a trained teacher network, wherein the teacher network is a CricaVPR network optimized by the multi-level feature mixing unit;

[0023] In the second training stage, the trained teacher network is used to supervise the training of the global descriptor extraction network enhanced with cross-image correlation information, so as to obtain the trained global descriptor extraction network enhanced with cross-image correlation information.

[0024] In one embodiment, when training the teacher network and the global descriptor extraction network enhanced with cross-image correlation information, a small batch training method is adopted, and each batch of training data includes a plurality of corresponding sample images of different locations, wherein a preset number of sample images are set for each location;

[0025] The teacher network and the global descriptor extraction network enhanced with cross-image correlation information are trained once using a batch of training data.

[0026] In one embodiment, in the second stage:

[0027] After preprocessing each sample image in a batch of training data, the trained teacher network and the global descriptor extraction network enhanced with cross-image correlation information are respectively input to obtain the corresponding teacher prediction results and student prediction results;

[0028] Calculating a mean square error loss function based on the teacher's prediction results and the student's prediction results, and calculating a multi-similarity loss function based on the student's prediction results;

[0029] According to the two calculation results, the learnable parameters in the global descriptor extraction network for cross-image correlation information enhancement are adjusted until a specified number of iterations is reached to obtain the trained global descriptor extraction network for cross-image correlation information enhancement.

[0030] The present application also provides a visual place recognition device with enhanced cross-image association information, the device comprising:

[0031] The module for acquiring images for locations to be identified is used to acquire visual image data for locations to be identified;

[0032] A query global descriptor acquisition module is used to extract the query global descriptor of the visual image data using a trained global descriptor extraction network enhanced by cross-image correlation information, wherein the trained global descriptor extraction network enhanced by cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit, the multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature, the multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set, and the data in the regional feature set is rearranged and input into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature to obtain a stable and distinguishable query global descriptor;

[0033] A database global descriptor acquisition module, used to extract database global descriptors having location image data in a map database using the trained global descriptor extraction network enhanced with cross-image association information;

[0034] The place position identification module is used to identify the place position in the visual image data according to the similarity between the query global descriptor and the database global descriptor.

[0035] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0036] Acquiring visual image data for location recognition;

[0037] Utilize the trained global descriptor extraction network enhanced with cross-image correlation information to extract the query global descriptor of the visual image data, wherein the trained global descriptor extraction network enhanced with cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit, the multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature, the multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set, rearranges the data in the regional feature set and inputs it into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature, so as to obtain the stable and distinguishable query global descriptor;

[0038] Using the trained global descriptor extraction network enhanced with cross-image correlation information, extracting a database global descriptor having location image data in a map database;

[0039] Locations in the visual image data are identified based on similarities between the query global descriptor and database global descriptors.

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0041] Acquiring visual image data for location recognition;

[0042] Utilize the trained global descriptor extraction network enhanced with cross-image correlation information to extract the query global descriptor of the visual image data, wherein the trained global descriptor extraction network enhanced with cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit, the multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature, the multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set, rearranges the data in the regional feature set and inputs it into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature, so as to obtain the stable and distinguishable query global descriptor;

[0043] Using the trained global descriptor extraction network enhanced with cross-image correlation information, extracting a database global descriptor having location image data in a map database;

[0044] Locations in the visual image data are identified based on similarities between the query global descriptor and database global descriptors.

[0045] The above-mentioned visual place recognition method, device and equipment for cross-image association information enhancement extracts the query global descriptor of visual image data by using the trained global descriptor extraction network for cross-image association information enhancement, wherein the global descriptor extraction network for cross-image association information enhancement includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit and an image feature self-enhancement encoding unit. The multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature. The multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed features to obtain a regional feature set. The data in the regional feature set is rearranged and input into the image feature self-enhancement encoding unit for self-enhancement of each regional feature to obtain the query global descriptor with discrimination. At the same time, the trained global descriptor extraction network for cross-image association information enhancement is used to extract the database global descriptor of the location image data in the map database. Finally, according to the similarity between the query global descriptor and the database global descriptor, the location of the place in the visual image data is identified. This method can achieve stable and high-accuracy positioning for visual place recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of a flow chart of a visual place recognition method with enhanced cross-image association information in one embodiment;

[0047] Figure 2 A schematic diagram of general steps and test flow of a visual place recognition task in one embodiment;

[0048] Figure 3 A schematic diagram of the structure of a global descriptor extraction network for cross-image correlation information enhancement in one embodiment;

[0049] Figure 4 A schematic diagram of the structure of a multi-level feature mixing unit in one embodiment;

[0050] Figure 5 A schematic diagram of the structure of a multi-level regional feature aggregation unit in one embodiment;

[0051] Figure 6 A schematic diagram of the network structure of the teacher network Super-CricaVPR in one embodiment;

[0052] Figure 7 is a training flow chart of a teacher network in one embodiment;

[0053] Figure 8 A flowchart of training a global descriptor extraction network for cross-image correlation information enhancement in one embodiment;

[0054] Fig. 9 A schematic diagram showing the comparison of qualitative results of this method with other visual place recognition methods in an experiment;

[0055] Fig.10 A structural block diagram of a visual place recognition device with enhanced cross-image correlation information in one embodiment;

[0056] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] In view of the problems existing in the visual image place recognition method developed based on the large model DINOv2 in the prior art, as shown in 1, a visual place recognition method with cross-image correlation information enhancement is provided, which specifically includes the following steps:

[0059] Step S100, obtaining visual image data for location recognition.

[0060] Step S110, using the trained global descriptor extraction network for cross-image correlation information enhancement, extract the query global descriptor of the visual image data, wherein the trained global descriptor extraction network for cross-image correlation information enhancement includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit and an image feature self-enhancement encoding unit, the multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature, the multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed features to obtain a regional feature set, rearranges the data in the regional feature set and inputs it into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature, so as to obtain a stable and discriminative query global descriptor.

[0061] Step S120, using the trained global descriptor extraction network enhanced with cross-image correlation information, extracting the database global descriptor with location image data in the map database.

[0062] Step S130 , identifying the location in the visual image data based on the similarity between the query global descriptor and the database global descriptor.

[0063] In this embodiment, based on the conventional visual place recognition task framework, such as Figure 2 As shown in the figure, at the same time, by making full use of the rich visual information contained in the large model DINOv2 and the feature optimization brought by the cross-image information exchange encoder in the CricaVPR network, a global descriptor extraction network containing a multi-level feature mixing unit and a self-enhanced encoder with implicit image association information (i.e., image feature self-enhanced encoding unit) is proposed, namely, a global descriptor extraction network enhanced with cross-image association information. In this network, the multi-level feature mixing unit utilizes the multi-layer output features of DINOv2 and fully integrates the features in the channel dimension and spatial dimension respectively. In addition, the self-enhanced encoder distills the image association information explicitly implemented in the cross-image information exchange encoder into its own weight model, so that the output global features can be stably enhanced by the image association information.

[0064] Compared with existing visual place recognition methods, the method proposed in this application can make full use of the rich visual information contained in the large model DINOv2 to provide local features related to the visual place recognition task. Compared with the advanced method CricaVPR for visual place recognition tasks, the global descriptor extraction network in this method can output a global descriptor that is stably enhanced by image association information, achieving stable and high-accuracy positioning for visual place recognition tasks.

[0065] In step S100, the obtained visual image data to be used for location recognition may be one visual image or multiple visual images. When the visual image data is multiple visual images, the trained global descriptor extraction network enhanced with cross-image correlation information simultaneously extracts the query global descriptor of each visual image.

[0066] In this embodiment, the visual image is an RGB image.

[0067] like Figure 3 As shown in FIG. 1 , the data processing process of the global descriptor extraction network for cross-image correlation information enhancement is shown, wherein the network includes three processing units in sequence, namely a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit.

[0068] In this embodiment, in the multi-level feature mixing unit: after the visual image input to the multi-level feature mixing unit is divided into multiple image blocks, one-dimensional flattening and linear projection are performed to obtain a block vector sequence, a learnable vector is prepended to each block vector in the block vector sequence, and then an additional position code is superimposed on each vector to obtain a vector sequence, and then the vector sequence is input into the DINOv2 large model for feature extraction to obtain block features, and then the block features are processed in sequence using convolutional layers, ReLU activation functions and block element mixers to obtain multi-level mixed features.

[0069] Specifically, Figure 4 As shown, taking a single input graph as an example, the two-dimensional input graph x∈R H×W×C (length is H, width is W, C is the number of original channels), is first divided into N image blocks, then flattened in one dimension, and then transformed into block vectors through linear projection In addition, a learnable [class] vector is prepended to the block vector to form the vector To save the position information, an additional position code is added to the vector x0 The resulting vector It can be input into the subsequent DINOv2 model for feature extraction.

[0070] The large model DINOv2 consists of Transformer layers, each of which contains a multi-head self-attention module (MSA) and a multi-layer perceptron (MLP), and there is a layer normalization operation before these two modules and a residual connection operation after these two modules.

[0071] The formulas involved in the above operations are as follows:

[0072] z0=x0+E pos

[0073] z l ′=MSA(LN(z l-1 ))+z l-1

[0074] z l =MLP(LN(z l ′))+z l '

[0075] In the above formula, l=1,...,L.

[0076] Since the DINOv2 model is trained on a large-scale unsupervised dataset and contains rich visual feature information, the global descriptor extraction network for enhancing cross-image correlation information directly uses the DINOv2 model with frozen weights as the backbone network, and the network weights of DINOv2 are not updated in the subsequent model training process.

[0077] Feature z output by DINOv2 i Include Class Features and block features Both can be used to describe the global information of the image. To reduce the amount of calculation, the class features are discarded in this method and only the output block features are considered. In order to make full use of the features output by DINOv2, the block features output by the last M layers are connected in series in the global descriptor extraction network with enhanced cross-image correlation information. get First, a 1×1 convolution is used to fully integrate the channel information of the feature and reduce the channel dimension. Then, the ReLU activation function is used to enhance the nonlinearity of the feature and flatten the block feature for subsequent operations.

[0078] The formulas involved in the above operations are as follows:

[0079]

[0080] Y′=Concat(y L-M ;…;y L )

[0081] Y = Flatten(ReLU(Conv(Y′)))

[0082] In the above formula, i = LM, ..., L and

[0083] Furthermore, a block mixer is used to fully communicate and fuse the spatial information between N different blocks. The multilayer perceptron in the block mixer works independently on each channel, and its parameters are shared across all channels. The multilayer perceptron consists of two fully connected layers and a ReLU function, and the number of feature channels of the output is the same as the number of feature channels of the input. The formula involved in this process is as follows:

[0084] Y={Y j}

[0085] F j =Y j +W2σ(W1(LN(Y j ))

[0086] F={F j}

[0087] In the above formula, j = 1, ..., C2, σ represents the ReLU activation function, W2∈R N×P , W3∈R P×N And F is the output multi-level mixed feature.

[0088] In this embodiment, in the multi-level regional feature aggregation unit: the multi-level mixed features are rearranged into a two-dimensional feature map, based on the two-dimensional feature map, GeM pooling operations are performed in 14 regions at three different levels of 1×1, 2×2, and 3×3 to generate regional features of multiple dimensions. Finally, the regional features of multiple dimensions are connected in series to obtain a regional feature set, such as Figure 5 shown.

[0089] Specifically, the multi-level mixed features aggregate multi-level regional features in the aggregation layer, such as Figure 5 The multi-level mixed features are first rearranged into a two-dimensional feature map (length H, width W, number of channels C2), and then divided into 14 regions of three different levels of 1×1, 2×2, and 3×3 in the aggregation layer for GeM pooling operations to generate 14 regional features of dimension C2, which are then connected in series to obtain the output of the aggregation layer.

[0090] In this embodiment, the image feature self-enhancement encoding unit is composed of a basic structure of a Transformer network. At this time, taking the input visual image data as B visual images as an example, at this time, the regional feature set The rearrangement of is to let each regional feature enter the self-enhanced encoder one by one, the sample batch input to the encoder becomes 14B, the sequence size becomes 1, and the rearranged regional feature set is The self-enhanced encoder consists of a basic Transformer structure, in which 14B regional features can only be self-enhanced encoded in the multi-head self-attention module. The self-enhanced encoded regional features can be rearranged and L2 normalized to form a stable and discriminative global descriptor

[0091] In this embodiment, a training method for the global descriptor extraction network for cross-image correlation information enhancement is also provided, including two training stages: in the first training stage, the teacher network is first trained to obtain a trained teacher network, wherein the teacher network is a CricaVPR network optimized by a multi-level feature mixing unit. In the second training stage, the trained teacher network is used to supervise the training of the global descriptor extraction network for cross-image correlation information enhancement, and a trained global descriptor extraction network for cross-image correlation information enhancement is obtained.

[0092] Specifically, the training process of SciceVPR (global descriptor extraction network with enhanced cross-image correlation information) proposed in this paper is divided into two stages. Taking a batch of B RGB images as an example, its training process is briefly described. First, the above multi-level feature mixing module is used to optimize the CricaVPR model structure to obtain the Super-CricaVPR model structure, as shown in the figure. Figure 6 As shown. The small batch training method is used to train them separately to obtain the Super-CricaVPR network model that can explicitly utilize cross-image communication information. The training process at this stage is as follows Figure 7 As shown; then initialize the SciceVPR network using multi-level feature hybrid units and self-enhanced encoders, use the trained Super-CricaVPR as the teacher network, supervise the small batch training process of the student network SciceVPR, and obtain the SciceVPR network model with implicit image association information. The training process at this stage is as follows Figure 8 shown.

[0093] Furthermore, in order to enable the global descriptor extraction network enhanced with cross-image correlation information to learn the feature optimization brought by the cross-image information exchange encoder in Super-CricaVPR, in this embodiment, the global descriptor extraction network enhanced with cross-image correlation information is trained in the form of Super-CricaVPR as a teacher network. At the same time, based on the original architecture of CricaVPR, Super-CricaVPR replaces the fine-tuned large model DINOv2 with the above-mentioned multi-level feature mixing unit.

[0094] Furthermore, when training the teacher network and the global descriptor extraction network enhanced with cross-image correlation information, a small batch training method is adopted. Each batch of training data includes multiple corresponding sample images of different locations, wherein a preset number of sample images are set for each location. The teacher network and the global descriptor extraction network enhanced with cross-image correlation information are trained once using a batch of training data.

[0095] In this embodiment, when using the training data to train the teacher network and the global descriptor extraction network with enhanced cross-image correlation information, the training data is also preprocessed, the sample image resolution in the training data is uniformly scaled to 224×224, and then the three random data enhancement methods provided in the torchvision.transforms.RandAugment function are used. If interpolation operation is required, bilinear interpolation is used, and finally the image is normalized using the mean and standard deviation of the ImageNet dataset.

[0096] Furthermore, in the teacher network, the first two units, the multi-level feature mixing unit and the multi-level regional feature aggregation unit, have been described in the previous article and will not be repeated here. In the cross-image information exchange encoder, the regional feature set output by the multi-level regional feature aggregation unit is encoded for information exchange. This stage is introduced by taking the number of batch samples as B as an example. The most critical step in realizing cross-image information exchange encoding is to encode the regional feature set. That is, all x-th regional features in a batch B are connected in series and input into the cross-image information exchange encoder in a sequence. At this time, the sample batch input into the encoder becomes 14, the sequence size becomes B, and the rearranged regional feature set is

[0097] Specifically, the cross-image information exchange encoder consists of two basic Transformer structures, in which all regional features in sequence B can communicate with each other in the multi-head self-attention module. The regional features after cross-image information exchange can form a discriminative global descriptor by rearranging and L2 normalization.

[0098] Furthermore, in the first training stage, the Super-CricaVPR model (teacher model) was trained using a small batch training method, with a batch size of 288, including images from 72 different locations (4 images per location). The multi-similarity loss function and online difficult sample mining strategy were used to find B1 positive sample pairs (two sample images of the same location, but with low global descriptor similarity) and B2 negative sample pairs (two sample images of different locations, but with high global descriptor similarity) that were difficult to distinguish among the 288 images, and the multi-similarity loss function was calculated. The network parameters were then updated through back propagation. The training was stopped after the specified number of iterations was reached, otherwise the training continued. The calculation of the multi-similarity loss function is expressed as:

[0099]

[0100] In formula (1), B is the batch size 288, for each query image I q , is the set of positive sample index p, is the set of negative sample index n. pq and S qn They are positive sample pairs {I q ,I p} and negative sample pairs {I q ,I n}, α, β and λ are hyper parameters.

[0101] In this embodiment, in the second stage: after preprocessing each sample image in a batch of training data, the trained teacher network and the global descriptor extraction network enhanced with cross-image correlation information are respectively input, and the corresponding teacher prediction results and student prediction results are obtained, the mean square error loss function is calculated according to the teacher prediction results and the student prediction results, and the multi-similarity loss function is calculated according to the student prediction results, and the learnable parameters in the global descriptor extraction network enhanced with cross-image correlation information are adjusted according to the two calculation results until convergence, so as to obtain a trained global descriptor extraction network enhanced with cross-image correlation information.

[0102] Specifically, the input image first undergoes the same image preprocessing as the training Super-CricaVPR, and then enters the trained Super-CricaVPR network model and the SciceVPR network to be trained. The multi-level hybrid feature extraction and multi-level regional feature aggregation stages experienced in the SciceVPR network model are exactly the same as those in Super-CricaVPR. The difference lies in the image feature self-enhancement encoding stage in SciceVPR, see Figure 3 The operations involved after the aggregation layer and the image feature self-enhancement coding unit have been explained above and will not be explained here.

[0103] Furthermore, the training of the SciceVPR network model also adopts a small batch training method, with a batch size of 288, including images from 72 different locations (4 images per location), and also adopts a multi-similarity loss function and an online difficult sample mining strategy. The difference is that the supervision signal of the Super-CricaVPR teacher network is added during the loss calculation process. The global descriptor generated by SciceVPR is made as similar as possible to the global descriptor generated by Super-CricaVPR, so that the weights of the self-enhanced encoder in SciceVPR contain cross-image correlation information, and the mean square error function shown in formula (2) is used for loss calculation. The mean square error function is expressed as:

[0104]

[0105] Next, according to formula (1) and formula (2), the total loss function for training the SciceVPR network is obtained, which is expressed as:

[0106] L 总 =γL1+ηL2 (3)

[0107] In formula (3), γ and η are hyperparameters.

[0108] Then, the SciceVPR network parameters are updated through back propagation. The training is stopped after the specified number of iterations is reached, otherwise the training continues. In this process, Super-CricaVPR uses the trained network parameters and the parameters are fixed during the entire SciceVPR training phase without parameter updates.

[0109] In steps S120 and S130, the data with specific location information in the database is input into the trained global descriptor extraction network enhanced with cross-image correlation information to obtain the database global descriptor. The location of the place in the visual image is then determined by calculating the distance between the query global descriptor and the database global descriptor.

[0110] In order to evaluate the performance of the global descriptor extraction network with enhanced cross-image correlation information proposed in this paper, the model was trained on the public dataset GSV-Cities, which is widely used in visual place recognition tasks, and tested on other public datasets Pitts30k-test, Tokyo24 / 7, MSLS-val / challenge, AmsterdamTime and SVOX datasets. The main features of these datasets are shown in Table 1.

[0111] Table 1 Overview of the visual place recognition test dataset used in the experiment

[0112]

[0113] The Pitts30k dataset contains 10,000 database images downloaded from Google Street View, and each train / val / test dataset has GPS tags. The experiment evaluates the model on the test dataset, which contains 6,816 query images that are taken at different times and years than the database images.

[0114] The Tokyo24 / 7 dataset consists of 75,984 database images from Google Street View and 315 query images taken with mobile phones mainly on sidewalks, both of which have GPS tags. The database images are all daytime images, while the query images can be daytime or nighttime.

[0115] MSLS is a large dataset for urban and suburban visual place recognition tasks, containing image sequences spanning nine years. MSLS covers the following challenges of visual place recognition: seasonal and weather changes, lighting changes at different times of the day, and different viewpoints. Images in MSLS have GPS tags. Models are tested on a public validation set (MSLS-val) and a test set with metadata preserved (MSLS-challenge).

[0116] The AmsterTime dataset contains 1231 pairs of images of urban areas. Each pair includes a historical grayscale query image and a corresponding modern RGB reference image, which represent the same location. This dataset is challenging due to domain variations such as viewpoint, modality (RGB vs. grayscale), and very long time span.

[0117] SVOX is a cross-domain dataset for evaluating visual place recognition models under various weather conditions. The dataset covers the city of Oxford, the dataset images come from Google Street View, and the query images come from the Oxford RobotCar dataset, which covers various weather and lighting conditions such as cloudy, rainy, sunny, snowy, and night.

[0118] Test process: First, the database image and query image are preprocessed (in the test phase, only the images need to be uniformly scaled to 224×224, and the mean and standard deviation of the ImageNet dataset are used to normalize the images). The database image is generated by the SciceVPR network model and stored as a database global descriptor. The query image is generated by the SciceVPR network model as a query global descriptor. The database image that is most similar to the query image is found through the nearest neighbor search method, and the geographical location where the database image is stored is used as the location of the query image. Figure 2 shown.

[0119] Evaluation criteria: Recall@N is an evaluation criterion for visual place recognition tasks, which is defined as the percentage of query images among all query images for which at least one of the top N predictions is from the same place as the query image, simplified as R@N. For the GPS-tagged Pitts30k, Tokyo24 / 7, MSLS, and SVOX datasets, if the predicted database image is within 25 meters of the query image, they are considered to be from the same place; AmsterTime is a collection of image pairs where only the database image corresponding to the query image is from the location of the query image.

[0120] Experimental implementation details: Super-CricaVPR and SciceVPR were trained on two NVIDIA GeForce RTX 3090s, with a batch size of 288 (images from 72 different locations in GSV-Cities, each location contains 4 images), the DINOv2 backbone network used was the ViT-B / 14 model, the output features of the last four layers of DINOv2 were concatenated, and the number of feature channels after concatenation was reduced to 768 through 1×1 convolution, the number of block mixers was 2 and the hidden dimension was 16×16, and the global descriptor dimensions of Super-CricaVPR and SciceVPR were both 14×768, and the global descriptor output by SciceVPR was further reduced to 4096 through PCA. The hyperparameters in formula (1) are α=1, β=50, λ=0; the hyperparameters in formula (3) are γ=1, η=1. The model was trained using the Adam optimizer, with the initial learning rate set to 0.0001 and multiplied by 0.5 after every 3 training rounds. The number of training rounds for Super-CricaVPR was set to 10 rounds, and the number of training rounds for SciceVPR was set to 2 rounds.

[0121] Experimental comparison: The experiment of this method compares the test results of the baseline method and this method on the visual place recognition dataset containing various challenges in the past three years. Since the test results of the CricaVPR model are greatly affected by the arrangement of the dataset and the batch size of the input image, for the sake of fairness, the batch size of the CricaVPR model is set to 1 during the test, named CricaVPR-single. The comparison results are shown in Tables 2 and 3. In Tables 2 and 3, R@1, R@5, and R@10 are the evaluation criteria for accuracy. The best results are indicated in bold, and the second best results are indicated by underscores.

[0122] Table 2 Comparison results of this method with other mainstream state-of-the-art visual place recognition models on various benchmark datasets

[0123]

[0124] Table 3 Comparison results of this method with other mainstream state-of-the-art visual place recognition models on various challenging test datasets

[0125]

[0126] As can be seen from Table 2, SciceVPR surpasses all other models in terms of Recall@1 and Recall@10, especially on the Tokyo24 / 7 dataset, where Recall@1 is as high as 94.9%. Tokyo24 / 7 contains significant day / night changes and perspective changes, and this result shows the robustness of the SciceVPR model to these changes. In addition, SciceVPR also shows good performance on the MSLS-val / challenge dataset, which has a long time span and contains some suburban scenes, showing the effectiveness of our method.

[0127] Table 3 further demonstrates the generalization of the SciceVPR model. On the AmsterTime dataset with modal changes and long-term changes, and on the SVOX-related datasets with various weather changes, the SciceVPR model almost achieved the best place recognition results, and on the AmsterTime, SVOX-Night, and SVOX-Sun datasets, Recall@1 surpassed the second-ranked CricaVPR-single by 9.4%, 11.5%, and 6.6%, respectively. In summary, SciceVPR has excellent performance in various visual place recognition scenarios.

[0128] Fig. 9 The positioning effect of SciceVPR is qualitatively demonstrated, where the green box represents the correct location recognition result and the red box represents the incorrect location recognition result. Fig. 9 We present some difficult positioning scenarios for visual place recognition tasks, including large-angle viewpoint changes, lighting changes between day and night, weather changes within a year, occlusions on buildings, and structural changes in the same location over time. The results show that SciceVPR can still correctly identify locations in these difficult scenarios, while other methods can only identify similar images in different locations.

[0129] In the above-mentioned visual place recognition method enhanced by cross-image association information, by mining the rich visual information contained in the large model DINOv2 and using a self-enhanced encoder with implicit image association information, a stable and highly discriminative global descriptor is provided, which can achieve stable and high-accuracy positioning of the visual place recognition task. This method first preprocesses the input query image and database image, and then extracts multi-level mixed features, aggregates multi-level regional features, and self-enhances the image features in the SciceVPR network model, outputs a global descriptor implicitly enhanced by the image association information, and then uses the nearest neighbor search method to find the database descriptor that is closest to the query global descriptor. The geographical location of the database image is the location of the query image. This method can significantly improve the positioning accuracy of the visual place recognition task.

[0130] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0131] In one embodiment, Fig.10 As shown, a visual place recognition device with cross-image association information enhancement is provided, comprising: a place recognition image acquisition module 200, a query global descriptor acquisition module 210, a database global descriptor acquisition module 220 and a place position recognition module 230, wherein:

[0132] The image acquisition module 200 for obtaining the image data for the location to be identified is used to obtain the visual image data for the location to be identified;

[0133] A query global descriptor acquisition module 210 is used to extract the query global descriptor of the visual image data using a trained global descriptor extraction network enhanced by cross-image correlation information, wherein the trained global descriptor extraction network enhanced by cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit, the multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature, the multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set, and the data in the regional feature set is rearranged and input into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature to obtain a stable and distinguishable query global descriptor;

[0134] A database global descriptor acquisition module 220 is used to extract database global descriptors having location image data in a map database using the trained global descriptor extraction network enhanced with cross-image correlation information;

[0135] The place position identification module 230 is used to identify the place position in the visual image data according to the similarity between the query global descriptor and the database global descriptor.

[0136] For the specific limitations of the visual place recognition device with cross-image association information enhancement, please refer to the limitations of the visual place recognition method with cross-image association information enhancement above, which will not be repeated here. Each module in the above-mentioned visual place recognition device with cross-image association information enhancement can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0137] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a visual place recognition method with enhanced cross-image association information is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0138] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0140] Acquiring visual image data for location recognition;

[0141] Utilize the trained global descriptor extraction network enhanced with cross-image correlation information to extract the query global descriptor of the visual image data, wherein the trained global descriptor extraction network enhanced with cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit, the multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature, the multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set, rearranges the data in the regional feature set and inputs it into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature, so as to obtain the stable and distinguishable query global descriptor;

[0142] Using the trained global descriptor extraction network enhanced with cross-image correlation information, extracting a database global descriptor having location image data in a map database;

[0143] Locations in the visual image data are identified based on similarities between the query global descriptor and database global descriptors.

[0144] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0145] Acquiring visual image data for location recognition;

[0146] Utilize the trained global descriptor extraction network enhanced with cross-image correlation information to extract the query global descriptor of the visual image data, wherein the trained global descriptor extraction network enhanced with cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit, the multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature, the multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set, rearranges the data in the regional feature set and inputs it into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature, so as to obtain the stable and distinguishable query global descriptor;

[0147] Using the trained global descriptor extraction network enhanced with cross-image correlation information, extracting a database global descriptor having location image data in a map database;

[0148] Locations in the visual image data are identified based on similarities between the query global descriptor and database global descriptors.

[0149] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0150] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A visual place recognition method with enhanced cross-image correlation information, characterized in that: The method comprises: Acquiring visual image data for location recognition; Utilize the trained global descriptor extraction network enhanced with cross-image correlation information to extract the query global descriptor of the visual image data, wherein the trained global descriptor extraction network enhanced with cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit, the multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature, the multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set, rearranges the data in the regional feature set and inputs it into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature, so as to obtain the stable and distinguishable query global descriptor; Using the trained global descriptor extraction network enhanced with cross-image correlation information, extracting a database global descriptor having location image data in a map database; Locations in the visual image data are identified based on similarities between the query global descriptor and database global descriptors.

2. The visual place recognition method with enhanced cross-image correlation information according to claim 1, characterized in that: The visual image data is one visual image or multiple visual images. When the visual image data is multiple visual images, the trained global descriptor extraction network enhanced with cross-image correlation information simultaneously extracts query global descriptors of each visual image.

3. The visual place recognition method with enhanced cross-image correlation information according to claim 2, characterized in that: In the multi-level feature mixing unit: After dividing the visual image input to the multi-level feature mixing unit into a plurality of image blocks, one-dimensional flattening and linear projection are performed to obtain a block vector sequence; Prepending a learnable vector to each block vector in the block vector sequence, and then superimposing an additional position code on each vector to obtain a vector sequence; Inputting the vector sequence into the DINOv2 large model to extract features to obtain block features; The block features are processed in sequence using a convolutional layer, a ReLU activation function, and a block element mixer to obtain the multi-level mixed features.

4. The visual place recognition method with enhanced cross-image correlation information according to claim 2, characterized in that: In the multi-level regional feature aggregation unit: Rearranging the multi-level mixed features into a two-dimensional feature map; Based on the two-dimensional feature map, GeM pooling operations are performed in 14 regions at three different levels: 1×1, 2×2, and 3×3, to generate regional features of multiple dimensions; After the regional features of the multiple dimensions are connected in series, the regional feature set is obtained.

5. The visual place recognition method with enhanced cross-image correlation information according to claim 2, characterized in that: The image feature self-enhancement encoding unit is composed of a basic Transformer network structure.

6. The method for visual place recognition with enhanced cross-image association information according to any one of claims 1 to 5, characterized in that: When training the global descriptor extraction network for cross-image correlation information enhancement, two training stages are included: In the first training stage, the teacher network is first trained to obtain a trained teacher network, wherein the teacher network is a CricaVPR network optimized by the multi-level feature mixing unit; In the second training stage, the trained teacher network is used to supervise the training of the global descriptor extraction network enhanced with cross-image correlation information, so as to obtain the trained global descriptor extraction network enhanced with cross-image correlation information.

7. The visual place recognition method with enhanced cross-image correlation information according to claim 6, characterized in that: When training the teacher network and the global descriptor extraction network enhanced with cross-image correlation information, a small batch training method is adopted, and each batch of training data includes a plurality of corresponding sample images of different locations, wherein a preset number of sample images are set for each location; The teacher network and the global descriptor extraction network enhanced with cross-image correlation information are trained once using a batch of training data.

8. The visual place recognition method with enhanced cross-image association information according to claim 7, characterized in that: In this second phase: After preprocessing each sample image in a batch of training data, the trained teacher network and the global descriptor extraction network enhanced with cross-image correlation information are respectively input to obtain the corresponding teacher prediction results and student prediction results; Calculating a mean square error loss function based on the teacher's prediction results and the student's prediction results, and calculating a multi-similarity loss function based on the student's prediction results; According to the two calculation results, the learnable parameters in the global descriptor extraction network for cross-image correlation information enhancement are adjusted until a specified number of iterations is reached to obtain the trained global descriptor extraction network for cross-image correlation information enhancement.

9. A visual place recognition device with enhanced cross-image correlation information, characterized in that: The device comprises: The module for acquiring images for locations to be identified is used to acquire visual image data for locations to be identified; A query global descriptor acquisition module is used to extract the query global descriptor of the visual image data using a trained global descriptor extraction network enhanced by cross-image correlation information, wherein the trained global descriptor extraction network enhanced by cross-image correlation information includes a multi-level feature mixing unit, a multi-level regional feature aggregation unit, and an image feature self-enhancement encoding unit, the multi-level feature mixing unit uses the DINOv2 large model to perform multi-level feature extraction on the visual image data, and fuses the extracted multi-level features in the channel dimension and the spatial dimension to obtain a multi-level mixed feature, the multi-level regional feature aggregation unit performs multi-level regional feature aggregation on the multi-level mixed feature to obtain a regional feature set, and the data in the regional feature set is rearranged and input into the image feature self-enhancement encoding unit to perform self-enhancement of each regional feature to obtain a stable and distinguishable query global descriptor; A database global descriptor acquisition module, used to extract database global descriptors having location image data in a map database using the trained global descriptor extraction network enhanced with cross-image association information; The place position identification module is used to identify the place position in the visual image data according to the similarity between the query global descriptor and the database global descriptor.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images