A scene recognition method based on sample mining strategy and position consistency verification

Abstract

The application provides a scene recognition method based on sample mining strategy and position consistency verification, and relates to the technical fields of computer vision and robots. The method specifically comprises: acquiring a query image and a database image and inputting the pre-trained deep neural network for feature extraction, and then calculating the global distance and the local distance of the query image and the database image; based on the global distance and the local distance of the query image and the database image, the best positive sample and the absolute negative sample of the query image are acquired, and a training triple is constructed to train the deep neural network to obtain the final deep neural network; the obtained deep neural network is used to re-extract the global features of the query image and calculate the global distance, and then K candidate frame images are selected from the database image and a reordering algorithm based on position consistency verification is used for screening to obtain the final matching image, so as to improve the matching precision of the scene image in the visual scene recognition task.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and robotics, and in particular to a scene recognition method based on sample mining strategies and position consistency verification. Background Technology

[0002] Visual scene recognition technology refers to determining whether a robot has visited its current scene before based on visual information acquired from the environment. This technology plays a crucial role in mobile robot localization tasks. For example, in Simultaneous Localization and Mapping (SLAM), accurate scene recognition helps the robot determine whether it is already in a previously visited area, thus enabling loop closure detection and map optimization. This is essential for ensuring map consistency and reducing accumulated errors. In current research, visual scene recognition is often considered an image retrieval task. The robot solves the scene recognition problem by retrieving the single image most similar to the task from an image database. This is typically achieved through target detection and analysis within the scene, or by performing stable feature extraction to analyze and assess the current scene.

[0003] The paper "IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14141–14152, 2021" proposes a novel hierarchical scene recognition retrieval architecture. This architecture retrieves candidate frames of query images using the NetVLAD method, then aggregates and matches local features defined on a feature space grid, and further introduces local features with complementary scales through an integral feature space for multi-scale fusion, demonstrating that the fused features are highly robust to appearance and viewpoint changes. However, the re-ranking module in this architecture neglects real-time issues, resulting in extremely long computation times and low model efficiency. The paper "IEEE Robotics and Automation Letters, 2684-2691, 2022" proposes a scene recognition method based on a tightly coupled training strategy. This method combines global features and local feature distance ranking for positive sample mining, uses local feature distance to re-rank samples to select the best positive samples, and utilizes a tightly coupled training strategy to continuously optimize the selection of positive samples, making it more beneficial for model training. However, this method only considers the local feature distance in the sample, which has a large computational burden, and at the same time ignores the overall similarity of the sample images. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a scene recognition method based on sample mining strategies and position consistency verification by coupling a global feature-based ranking algorithm and a local feature-based re-ranking algorithm during the learning and training process of deep networks. This method achieves the coupling of global and local features and is applied to Visual Place Recognition (VPR) tasks to improve scene image matching accuracy, aiming to solve the problems of long processing time and low efficiency in existing re-ranking techniques.

[0005] This invention proposes a scene recognition method based on sample mining strategy and location consistency verification, which includes the following steps:

[0006] Step 1: Obtain the query image q, the database image R, and the pre-trained deep neural network. Input the query image q and the database image R into the deep neural network to obtain the local and global features of the query image q and the local and global features of the database image R, respectively. Then calculate the global distance and local distance between the query image q and the database image R.

[0007] Step 2: Based on the global and local distances between the query image q and the database image R, a sample mining strategy is used to obtain the best positive sample of the query image q, and a global matcher is used to obtain the absolute negative sample of the query image q. The query image q, the best positive sample, and the absolute negative sample are used as training triples to train the deep neural network, and the fusion loss function is used to obtain the final deep neural network.

[0008] Step 3: Use the deep neural network obtained in Step 2 to re-extract global features from the query image q and calculate the global distance. Then select K candidate frame images from the database images. Use a re-ranking algorithm based on position consistency verification to filter the candidate frame images and obtain the final matching image of the query image.

[0009] Step 1 further includes:

[0010] Step 1.1: Obtain the query image q and the database image R. Use a unified feature extractor as a pre-trained deep neural network. Input the query image q and the database image R into the unified feature extractor for feature extraction. Obtain the local features of the query image q and the database image R respectively and transform them into B×C dimensional local feature sequences; where B represents the number of local features; C represents the feature dimension of the local feature sequence.

[0011] Step 1.2: Based on the unified feature extractor, generalized mean GeM pooling is used to aggregate local feature sequences into global features;

[0012] Step 1.3: Calculate the global distance D between the query image q and the database image R using L2 distance based on global features. g (q,R), the local distance D between the query image q and the database image R is calculated using the bidirectional dynamic time warping (DTW) algorithm based on the local feature sequence. l (q,R);

[0013] Step 2 further includes:

[0014] Step 2.1: Select the best positive sample p of the query image q using a global-local reordering mining method based on the random sample consensus RANSAC algorithm. * ;

[0015] Step 2.2: Optimal positive sample p based on query image q * A global matcher is used to select j absolute negative samples n for the query image q. j ;

[0016] Step 2.3: Utilize the query image q and the best positive sample p * and j absolute negative samples n j Construct a training triplet, divide the resulting training triplets into a training set and a validation set, and use the training set to train the unified feature extractor.

[0017] Step 2.4: Based on the best positive sample p * and absolute negative samples n j Define the global distance loss function and the local distance loss function, and calculate the global distance loss and the local distance loss obtained by training the unified feature extractor;

[0018] Step 2.5: Construct a fusion loss function by weighted summation of the global distance loss function and the local distance loss function, and calculate the fusion loss obtained by training the unified feature extractor based on the fusion loss function;

[0019] Step 2.6: Repeat steps 2.4-2.5 until the unified feature extractor meets the preset training results or number of iterations, then complete the training and obtain the trained unified feature extractor;

[0020] Step 2.7: Validate the trained unified feature extractor using the validation set to obtain the final unified feature extractor;

[0021] Step 2.1 describes using a global-local reordering mining method based on the RANSAC algorithm to select the best positive sample p of the query image q. *The process is as follows: Based on the GPS tag of the query image q, determine the potential positive sample set P from the database image R; sort the images in the potential positive sample set P according to the global distance between the query image q and the database image R in ascending order; and select the first i potential positive samples p based on the sorting result. i As candidate samples for mining, the RANSAC algorithm based on local features is used to calculate the image similarity between the candidate samples and the query image q. Based on the calculated image similarity, i potential positive samples p are selected. i The best positive sample p for the query image q is selected from the data. * ;

[0022] The selection of the first i potential positive samples based on the sorting results is expressed as follows:

[0023]

[0024] Where p i arg min represents the first i samples that are globally closest to the query image q; i () indicates the index position of the i-th minimum value in the set; D g (q,P) represents the global distance between the query image q and the set of potential positive samples P;

[0025] The optimal positive sample p * Represented as:

[0026] p * =argmax{RANSAC(q,p i )}

[0027] Where RANSAC(q,p) i This indicates that the RANSAC algorithm is used to calculate the query image q and the potential positive sample p. i Image similarity between; argmax{·} is the maximum value function;

[0028] Step 2.1 involves selecting j absolute negative samples n. j The method is as follows: Determine the exact negative sample set N of the query image q from the database image R based on the GPS tag of the query image q; and determine the optimal positive sample p of the query image q based on the global distance between the query image q and the database image R. * Select j absolute negative samples n from the exact negative sample set N in ascending order of global distance. j At the same time, make the absolute negative sample n j All satisfy:

[0029] D g (q,p * )+mD g (q,nj )>0

[0030] Where D g (q,p * ) represents the query image q and the best positive sample p. * global distance; D g (q,n j ) represents the query image q and the absolute negative samples n j The global distance; m represents the set threshold;

[0031] The global distance loss function mentioned in step 2.4 is:

[0032] L g =∑ j max{(D g (q,p * )+mD g (q,n j )),0}

[0033] Where L g Represents the global loss value;

[0034] The local distance loss function is:

[0035] L l =∑ j max{(D l (q,p * )+mD l (q,n j )),0}

[0036] Where L l D represents the local loss value; l (q,p * ) represents the query image q and the best positive sample p. * Local distance; D l (q,n j ) represents the query image q and the absolute negative samples n j Local distance;

[0037] The fusion loss function mentioned in step 2.5 is:

[0038] L T =L g +λ*L l

[0039] Where L T λ represents the fusion loss value; λ represents the weight of the local distance loss function.

[0040] Step 3 further includes:

[0041] Step 3.1: Use the deep neural network obtained in Step 2 to re-extract global features from the query image q, and use the extracted global features to calculate the global distance between the query image q and the database image R as the global feature descriptor F. g ;

[0042] Step 3.2: Apply the global feature descriptor F g As a scene representation, the similarity between the query image q and the scene in the database image is calculated using cosine distance. Based on this similarity, the K images most similar to the query image q are retrieved from the image database as candidate frame images C.

[0043] Step 3.3: Perform local feature extraction on the query image q and K images in the candidate frame image C respectively to obtain the local feature maps f of the K candidate frame images. c and the local feature map f of the query image q q ;where f c and f q The size of each is n×n×D, where n×n represents the number of local features and D represents the feature dimension of the local feature map.

[0044] Step 3.4: For any image in the candidate frame images, generate the corresponding local feature map f. c and the local feature map f of the query image q q Perform bidirectional cosine similarity calculation, and determine the bidirectional nearest neighbor local feature matching set MNN based on the obtained bidirectional similarity matrix;

[0045] Step 3.5: For a specified local feature map f(m′) in the query image, a set of weight masks RV(m′) is obtained by averaging and normalizing along the channel dimension. The bidirectional nearest neighbor local feature matching set MNN is then filtered based on the weight mask RV(m′) to obtain the filtered bidirectional nearest neighbor local feature matching set MNN.

[0046] Step 3.6: Based on the local feature pairs (a,b) in the filtered bidirectional nearest neighbor local feature matching set MNN, calculate the position coordinates of a and b in the candidate frame image and the query image, respectively, and use Euclidean distance to calculate the distance between the above local feature pairs (a,b);

[0047] Step 3.7: Count the number of local feature pairs with a distance lower than the threshold t and use it as the verification score. Re-sort all images in the candidate frame images according to the verification scores of the query image, and select the image with the highest verification score as the final matching image.

[0048] The similarity between the query image q and the scene in the database image mentioned in step 3.2 is:

[0049]

[0050] Where s(·) represents the similarity between images; These represent the global feature descriptors of the queried image and the image in the database, respectively; ||·| represents the vector modulo operation;

[0051] The bidirectional nearest neighbor local feature matching set MNN mentioned in step 3.4 is:

[0052]

[0053] Where 'a' represents the label of the successfully matched local feature in the local features of the candidate frame image after the bidirectional matching operation between the candidate frame image and the query image, and a∈n×n×K; b represents the label of the successfully matched local feature in the local features of the query image after the bidirectional matching operation between the candidate frame image and the query image, and b∈n×n. Represents any local feature in the candidate frame image; This indicates a specified local feature in the query image; Represents a specified local feature in a candidate frame image; s(·) represents any local feature in the query image; s(·) represents the similarity measure.

[0054] The beneficial effects of adopting the above technical solution are as follows:

[0055] The present invention proposes a fast verification algorithm based on positional consistency. This algorithm verifies the positional information of local feature pairs, filters local descriptors, and reorders them, thereby maintaining higher matching accuracy in a very short processing time.

[0056] The fast positive sample mining strategy based on Random Sample Consensus (RANSAC) proposed in this invention can combine global and local features to mine the best positive samples, guide the deep neural network to learn features that are more conducive to location identification, and thus improve performance without adding extra burden.

[0057] This invention proposes a scene recognition method based on a fast positive sample mining strategy and position consistency verification. Within a hierarchical framework, it completes tightly coupled positive sample mining and candidate image re-ranking. This method utilizes the consistency of local feature position information in candidate frames as the basis for re-ranking, achieving accurate visual scene recognition. This improves the accuracy of robot visual positioning in large-scale scenes and facilitates more intelligent visual navigation and other tasks. Attached Figure Description

[0058] Figure 1This is a flowchart of a scene recognition method based on a fast positive sample mining strategy and location consistency verification in an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram illustrating feature extraction in an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the image retrieval principle based on positive sample mining provided in the embodiments of the present invention;

[0061] Figure 4 This is a schematic diagram illustrating the principle of scene recognition and retrieval provided in the embodiments of the present invention.

[0062] Figure 5 This is a schematic diagram of the principle of position consistency verification provided in the embodiments of the present invention;

[0063] Figure 6 This is a schematic diagram showing the comparison of time consumption and performance results provided in the embodiments of the present invention. Detailed Implementation

[0064] To facilitate understanding of this application, specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments. The following embodiments are illustrative of the invention but are not intended to limit its scope. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0065] The core idea of ​​this invention is as follows: Based on global and local features, a re-ranking algorithm is used to obtain the best positive sample of the query image q, and a global selector is used to obtain the absolute negative sample of image q. A training set for training a deep neural network is constructed based on the query image q, the best positive sample, and the absolute negative sample. By defining a fusion loss function, the coupling of global and local features is achieved, thereby enabling mutual promotion between global and local features. This allows the deep neural network to learn more effective information representation methods and richer feature details from the training set. Therefore, the consistency between global and local features is improved during training, resolving the conflict and confusion between the two in the results of visual location recognition tasks. The above learning strategy also improves the learning efficiency of the deep neural network and significantly reduces training time. Furthermore, the re-ranking algorithm, which performs position consistency verification in candidate frames, better leverages the advantages of spatial location verification in the retrieval results, significantly improving retrieval accuracy.

[0066] This embodiment presents a scene recognition method based on a fast positive sample mining strategy and location consistency verification, such as... Figure 1 As shown, the method includes the following steps:

[0067] Step 1: Obtain the query image q, the database image R, and the pre-trained deep neural network. Input the query image q and the database image R into the deep neural network to obtain the local and global features of the query image q and the database image R, respectively. Then, calculate the global distance and local distance between the query image q and the database image R.

[0068] In this embodiment, the query image q and the database image R are obtained from a publicly available training dataset as input images, and an open-source pre-trained deep neural network is obtained.

[0069] Step 1.1: Obtain the query image q and the database image R. Use a unified feature extractor as a pre-trained deep neural network. Input the query image q and the database image R into the unified feature extractor for feature extraction. Obtain the local features of the query image q and the database image R respectively and transform them into B×C dimensional local feature sequences; where B represents the number of local features and C represents the feature dimension of the local feature sequence.

[0070] In this embodiment, pre-trained compact convolutional transformers (CCTs) are used as the deep neural network to be learned, such as... Figure 2 As shown, the output image feature map is used as a local feature of the input image and converted into a 576×384-dimensional feature sequence.

[0071] Step 1.2: Based on the unified feature extractor, use Generalized Mean (GeM) pooling to aggregate local feature sequences into global features.

[0072] Step 1.3: Calculate the global distance D between the query image q and the database image R using L2 distance based on global features. g Given (q, R), the local distance D between the query image q and the database image R is calculated using the bidirectional Dynamic Time Warping (DTW) algorithm based on the local feature sequence. l (q,R).

[0073] In this embodiment, L2 distance is used to calculate the distance between the global features of the query image q and all images in the database image R. The global distance between the query image q and the database image R is:

[0074] D g (q,R)=||G q -G R ||2 (1)

[0075] Where D g(q,R) represents the global distance between the query image q and the database image R; G Q G represents the global features of the query image q; R Represents the global features of the database image R; ||·||2 represents the L2 distance operation.

[0076] In this embodiment, the bidirectional DTW algorithm is used to calculate the distance between the local feature sequences of the query image q and all images in the database image R. The local distance between the query image q and the database image R is denoted as D. l (q,R).

[0077] In this embodiment, step 1 involves feature extraction and feature distance calculation for the query image q and the database image R.

[0078] Step 2: Based on the global and local distances between the query image q and the database image R, a sample mining strategy is used to obtain the best positive sample of the query image q, and a global matcher is used to obtain the absolute negative sample of the query image q. The query image q, the best positive sample, and the absolute negative sample are used as training triples to train the deep neural network, and the final deep neural network is obtained by using the fusion loss function.

[0079] In this embodiment, the training datasets currently used in the field of scene recognition are all weakly labeled, meaning there are no explicit one-to-one data labels. Instead, for each query image q, there exists a potential positive sample set P and an exact negative sample set N. The potential positive sample set refers to the positive samples in P that best match the query image q, i.e., the positive samples at the same location, but it is impossible to determine which one in P this positive sample belongs to. The potential positive sample set may also contain negative random samples that do not match image q. Whether a match is found is determined based on the GPS tags in the dataset. Therefore, visual scene recognition tasks generally adopt a weakly supervised training strategy and perform training tuple mining and weakly supervised loss function definition during training.

[0080] In the training process of deep neural networks, the training objective is achieved through continuous iteration. The degree of objective completion is evaluated by the loss function, which is the gap between the expected output and the current output. The goal of training the network using training triples is to minimize the distance between the query image and positive samples, and maximize the distance between the query image and negative samples. The selection of positive and negative samples is generally achieved through sample mining algorithms.

[0081] Step 2.1: Select the best positive sample p of the query image q using a global-local reordering mining method based on the Random Sample Consensus (RANSAC) algorithm. * ;

[0082] The global-local reordering mining method based on the RANSAC algorithm is used to select the best positive sample p of the query image q. * The process is as follows: Based on the GPS tag of the query image q, determine the potential positive sample set P from the database image R; sort the images in the potential positive sample set P according to the global distance between the query image q and the database image R from smallest to largest; and select the first i potential positive samples p based on the sorting result. i As candidate samples for mining, the RANSAC algorithm based on local features is used to calculate the image similarity between the candidate samples and the query image q. Based on the calculated image similarity, i potential positive samples p are selected. i The best positive sample p for the query image q is selected from the data. * .

[0083] The selection of the first i potential positive samples based on the sorting results is expressed as follows:

[0084] p i =arg min i D g (q,P) (2)

[0085] Where p i arg min represents the first i samples that are globally closest to the query image q; i () indicates the index position of the i-th minimum value in the set; D g (q,P) represents the global distance between the query image q and the set of potential positive samples P.

[0086] The optimal positive sample p * Represented as:

[0087] p * =argmax{RANSAC(q,p i )} (3)

[0088] Where RANSAC(q,p) i This indicates that the RANSAC algorithm is used to calculate the query image q and the potential positive sample p. i Image similarity between the two; argmax{·} is the maximum value function.

[0089] In this embodiment, the conventional strategy directly calculates the global features of the image database based on the model parameters of the deep neural network in the current training phase, and selects the sample with the closest global distance from the potential positive sample set P of the query image q as the best positive sample p of the query image q. * .like Figure 3As shown, unlike traditional strategies, this implementation method uses a global-local reordering mining method based on the RANSAC algorithm as the positive sample mining strategy, which is divided into two stages: First, the global features of the image library are calculated based on the model parameters of the current training stage, and the five potential positive samples with the closest global distance are selected from the potential positive sample set P of the query image q according to formula (2); then, the image similarity between the potential positive samples and the query image q is calculated based on the RANSAC algorithm of local features as the basis for reordering, so as to select the sample p most similar to the query image q. * That is, to find the best positive sample of image q, which is used to construct the training tuple.

[0090] Step 2.2: Optimal positive sample p based on query image q * A global matcher is used to select j absolute negative samples n for the query image q. j .

[0091] The selection of j absolute negative samples n j The method is as follows: Determine the exact negative sample set N of the query image q from the database image R based on the GPS tag of the query image q; and determine the optimal positive sample p of the query image q based on the global distance between the query image q and the database image R. * Select j absolute negative samples n from the exact negative sample set N in ascending order of global distance. j At the same time, make the absolute negative sample n j All satisfy:

[0092] D g (q,p * )+mD g (q,n j )>0 (4)

[0093] Where D g (q,p * ) represents the query image q and the best positive sample p. * global distance; D g (q,n j ) represents the query image q and the absolute negative samples n j The global distance; m represents the set threshold; in this embodiment, m = 0.1 is set so that the global distance between the query image and the best positive sample is slightly smaller than the global distance between the query image and the absolute negative sample.

[0094] In this implementation, to reduce algorithm overhead, a negative sample mining method consistent with traditional strategies is adopted, namely a global matcher. This implementation selects the two images with the smallest distance to the query image from the exact negative sample set.

[0095] Step 2.3: Utilize the query image q and the best positive sample p * and j absolute negative samples n j Construct a training triplet, divide the resulting training triplets into a training set and a validation set, and use the training set to train the unified feature extractor.

[0096] Step 2.4: Based on the best positive sample p * and absolute negative samples n j Define the global distance loss function and the local distance loss function, and calculate the global distance loss and the local distance loss obtained by training the unified feature extractor.

[0097] The global distance loss function is:

[0098] L g =∑ j max{(D g (q,p * )+mD g (q,n j ),0} (5)

[0099] Where L g This represents the global loss value.

[0100] The local distance loss function is:

[0101] L l =∑ j max{(D l (q,p * )+mD l (q,n j ),0} (6)

[0102] Where L l D represents the local loss value; l (q,p * ) represents the query image q and the best positive sample p. * Local distance; D l (q,n j ) represents the query image q and the absolute negative samples n j The local distance.

[0103] In this implementation, the weakly supervised triplet loss function in the traditional strategy is:

[0104] L θ =∑ j max{(D g (q,p i )+mD g (q,n j )),0} (7)

[0105] Where L θ This represents the loss value of the weakly supervised triplet. It can be seen that the calculation ideas of formula (5) and formula (7) are essentially the same, only the best positive samples selected are different due to the different positive sample mining strategies. In addition, in order to guide the network to focus on local features during the training process, which is beneficial to the subsequent re-ranking operation, the loss function for local distance ranking is defined as formula (6).

[0106] Step 2.5: Construct a fusion loss function by weighted summation of the global distance loss function and the local distance loss function, and calculate the fusion loss obtained by training the unified feature extractor based on the fusion loss function;

[0107] The fusion loss function is:

[0108] L T =L g +λ*L l (8)

[0109] Where L T λ represents the fusion loss value; λ represents the weight of the local distance loss function.

[0110] In this embodiment, since the global and local distances between the query image q and the database image R are roughly equivalent, λ = 1 is chosen. Further constraints and joint optimization of the model are achieved by weighted summation of the global and local distance loss functions.

[0111] Step 2.6: Repeat steps 2.4-2.5 until the unified feature extractor meets the preset training results or number of iterations, then the training is complete and the trained unified feature extractor is obtained.

[0112] Step 2.7: Validate the trained unified feature extractor using the validation set to obtain the final unified feature extractor.

[0113] Step 3: Use the deep neural network obtained in Step 2 to re-extract global features from the query image q and calculate the global distance. Then select K candidate frame images from the database images. Use a re-ranking algorithm based on position consistency verification to filter the candidate frame images and obtain the final matching image of the query image.

[0114] In this embodiment, such as Figure 4As shown, a hierarchical architecture is adopted. The unified feature extractor obtained in step 2 is used to re-extract global features from the query image and the database image. The global distance between the query image and the database image is calculated based on the extracted global features. The database images are sorted from smallest to largest based on the global distance. The top-K candidate frame images that are close to the query image are selected based on the search sort. Then, the candidate frame images are used to complete the subsequent visual scene recognition and obtain the final localization result.

[0115] Step 3.1: Use the deep neural network obtained in Step 2 to re-extract global features from the query image q, and use the extracted global features to calculate the global distance between the query image q and the database image R as the global feature descriptor F. g .

[0116] Step 3.2: Apply the global feature descriptor F g As a scene representation, the similarity between the query image q and the scene in the database image is calculated using cosine distance. Based on this similarity, the K images most similar to the query image q are retrieved from the image database as candidate frame images C.

[0117] In this embodiment, the K images most similar to the query image q are denoted as C1 to C2. K .

[0118] The similarity between the query image q and the scene in the database image is:

[0119]

[0120] Where s(·) represents the similarity between images; These represent the global feature descriptors of the queried image and the image in the database, respectively; ||·| represents the vector modulo operation.

[0121] Step 3.3: Perform local feature extraction on the query image q and K images in the candidate frame image C respectively to obtain the local feature maps f of the K candidate frame images. c and the local feature map f of the query image q q ;where f c and f q The size of each is n×n×D, where n×n represents the number of local features and D represents the feature dimension of the local feature map.

[0122] In this embodiment, the first K candidate images C1 to C2 are... K Local feature extraction is performed between the query image q and the local feature map f, each with a size of 24×24×384. c and f q .

[0123] Step 3.4: For any image in the candidate frame images, generate the corresponding local feature map f. c and the local feature map f of the query image q q Perform bidirectional cosine similarity calculation, and determine the bidirectional nearest neighbor local feature matching set MNN based on the obtained bidirectional similarity matrix.

[0124] The bidirectional nearest neighbor local feature matching set MNN is:

[0125]

[0126] Where 'a' represents the label of the successfully matched local feature in the local features of the candidate frame image after the bidirectional matching operation between the candidate frame image and the query image, and a∈n×n×K; b represents the label of the successfully matched local feature in the local features of the query image after the bidirectional matching operation between the candidate frame image and the query image, and b∈n×n. Represents any local feature in the candidate frame image; This indicates a specified local feature in the query image; Represents a specified local feature in a candidate frame image; s(·) represents any local feature in the query image; s(·) represents the similarity measure.

[0127] Step 3.5: For a specified local feature map f(m′) in the query image, a set of weight masks RV(m′) is obtained by averaging and normalizing along the channel dimension. The bidirectional nearest neighbor local feature matching set MNN is then filtered based on the weight mask RV(m′) to obtain the filtered bidirectional nearest neighbor local feature matching set MNN.

[0128] The weight mask RV(m′) is:

[0129] RV(m′)=MinMaxNorm((f(m′))) (11)

[0130] MinMaxNorm(·) represents the normalization process.

[0131] In this embodiment, based on the local feature map f(m′) of a given image, a set of plug-and-play weight masks RV(m′) is obtained by averaging and normalizing along the channel dimension. When the calculated mask value is greater than a set threshold, the local feature map is retained as a local feature that is beneficial to scene matching. If it is less than the threshold, the weight mask is changed to 0, the local feature map is regarded as a local feature that is meaningless to scene matching, and it is deleted from the bidirectional nearest neighbor local feature matching set MNN.

[0132] Step 3.6: Based on the local feature pairs (a,b) in the filtered bidirectional nearest neighbor local feature matching set MNN, calculate the position coordinates of a and b in the candidate frame image and the query image, respectively, and use Euclidean distance to calculate the distance between the above local feature pairs (a,b);

[0133] In this embodiment, such as Figure 5 As shown, after obtaining the local feature pair (a,b) through bidirectional matching and filtering out meaningless local features, the position coordinates of a and b in the candidate image and the query image are calculated respectively, and the distance between the positions is obtained using Euclidean distance.

[0134]

[0135] Where (x,y) are the coordinates of the local feature in the candidate frame image, and (x′,y′) are the coordinates of the local feature in the query image.

[0136] Step 3.7: Count the number of local feature pairs with a distance lower than the threshold t and use it as the verification score. Re-sort all images in the candidate frame images according to their verification scores with the query image, and select the image with the highest verification score as the final matching image.

[0137] The verification score is:

[0138] Score = ∑distance <t (13)

[0139] The score represents the verification score.

[0140] In this embodiment, the verification score will be used as the basis for reordering the candidate frame images, and t will be set to 12.

[0141] To verify the effectiveness of the algorithm in visual scene recognition, this invention was tested on the St_Lucia dataset. Figure 6 As shown, based on experimental results, the algorithm proposed in this invention achieves a Recall@1 score of 0.992 and a Recall@5 score of 0.998 in environments with varying lighting conditions. These experimental results demonstrate that the method of this invention can effectively maintain high visual scene recognition accuracy in scenarios with rich lighting variations.

[0142] The reordering module based on positional consistency verification significantly impacts the time consumption of reordering-based visual scene recognition tasks. Generally, model accuracy improves with the performance of the reordering module. However, high-performance reordering modules lead to substantial computational overhead. Therefore, striking a balance between reordering module performance and computational efficiency is crucial. Our method was tested on the MSLS_val dataset and compared with other methods. By calculating the total time for feature extraction and matching, we found that while maintaining high accuracy, the time taken was only 0.028 seconds, which is more than an order of magnitude faster than existing methods. This means that our method achieves high accuracy in a very short time.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A scene recognition method based on sample mining strategy and location consistency verification, characterized in that, The method includes the following steps: Step 1: Obtain the query image q, the database image R, and the pre-trained deep neural network. Input the query image q and the database image R into the deep neural network to obtain the local and global features of the query image q and the local and global features of the database image R, respectively. Then calculate the global distance and local distance between the query image q and the database image R. Step 2: Based on the global and local distances between the query image q and the database image R, a sample mining strategy is used to obtain the best positive sample of the query image q, and a global matcher is used to obtain the absolute negative sample of the query image q. The query image q, the best positive sample, and the absolute negative sample are used as training triples to train the deep neural network, and the fusion loss function is used to obtain the final deep neural network. Step 3: Use the deep neural network obtained in Step 2 to re-extract global features from the query image q and calculate the global distance. Then select K candidate frame images from the database images. Use a re-ranking algorithm based on position consistency verification to filter the candidate frame images and obtain the final matching image of the query image. Step 3.1: Use the deep neural network obtained in Step 2 to re-extract global features from the query image q, and use the extracted global features to calculate the global distance between the query image q and the database image R as the global feature descriptor. ; Step 3.2: Convert the global feature descriptor As a scene representation, the similarity between the query image q and the scene in the database image is calculated using cosine distance. Based on this similarity, the K images most similar to the query image q are retrieved from the image database as candidate frame images C. Step 3.3: Perform local feature extraction on the query image q and the K images in the candidate frame image C respectively to obtain the local feature maps of the K candidate frame images. and the local feature map of the query image q ;in and The size of each is n×n×D, where n×n represents the number of local features and D represents the feature dimension of the local feature map. Step 3.4: For any image in the candidate frame images, generate the local feature map corresponding to that image. and the local feature map of the query image q Perform bidirectional cosine similarity calculation, and determine the bidirectional nearest neighbor local feature matching set MNN based on the obtained bidirectional similarity matrix; Step 3.5: For a specified local feature map in the query image A set of weight masks is obtained by averaging along the channel dimension and then normalizing. According to the weight mask The bidirectional nearest neighbor local feature matching set MNN is filtered to obtain the filtered bidirectional nearest neighbor local feature matching set MNN. Step 3.6: Match local feature pairs in the filtered bidirectional nearest neighbor local feature set MNN. ,calculate and The position coordinates in the candidate frame image and the query image are respectively used to calculate the above local feature pairs using Euclidean distance. The distance between them; Step 3.7: Statistically determine distances below the threshold The number of local feature pairs is used as the validation score. Based on the validation scores of all images in the candidate frame images and the query image, the images are reordered, and the image with the highest validation score is selected as the final matching image.

2. The scene recognition method based on sample mining strategy and location consistency verification according to claim 1, characterized in that, Step 1 further includes: Step 1.1: Obtain the query image q and the database image R. Use a unified feature extractor as a pre-trained deep neural network. Input the query image q and the database image R into the unified feature extractor for feature extraction. Obtain the local features of the query image q and the database image R respectively and transform them into B×C dimensional local feature sequences; where B represents the number of local features; C represents the feature dimension of the local feature sequence. Step 1.2: Based on the unified feature extractor, generalized mean GeM pooling is used to aggregate local feature sequences into global features; Step 1.3: Calculate the global distance between the query image q and the database image R using L2 distance based on global features. The local distance between the query image q and the database image R is calculated using the bidirectional dynamic time warping (DTW) algorithm based on the local feature sequence. .

3. The scene recognition method based on sample mining strategy and location consistency verification according to claim 2, characterized in that, Step 2 further includes: Step 2.1: Select the best positive sample for query image q using a global-local reordering mining method based on the random sample consensus RANSAC algorithm. ; Step 2.2: Optimal positive samples based on query image q A global matcher is used to select j absolute negative samples for the query image q. ; Step 2.3: Utilize the query image q and the best positive sample and j absolute negative samples Construct a training triplet, divide the resulting training triplets into a training set and a validation set, and use the training set to train the unified feature extractor. Step 2.4: Based on the best positive samples and absolute negative samples Define the global distance loss function and the local distance loss function, and calculate the global distance loss and the local distance loss obtained by training the unified feature extractor; Step 2.5: Construct a fusion loss function by weighted summation of the global distance loss function and the local distance loss function, and calculate the fusion loss obtained by training the unified feature extractor based on the fusion loss function; Step 2.6: Repeat steps 2.4-2.5 until the unified feature extractor meets the preset training results or number of iterations, then complete the training and obtain the trained unified feature extractor; Step 2.7: Validate the trained unified feature extractor using the validation set to obtain the final unified feature extractor.

4. The scene recognition method based on sample mining strategy and location consistency verification according to claim 3, characterized in that, Step 2.1 describes using a global-local reordering mining method based on the RANSAC algorithm to select the best positive sample for the query image q. The process is as follows: Based on the GPS tag of the query image q, determine the potential positive sample set P from the database images R; sort the images in the potential positive sample set P according to the global distance between the query image q and the database images R from smallest to largest; and select the first i potential positive samples based on the sorting result. As candidate samples for mining, the RANSAC algorithm based on local features is used to calculate the image similarity between the candidate samples and the query image q. Based on the calculated image similarity, i potential positive samples are selected. Filter out the best positive samples of the query image q. ; The selection of the first i potential positive samples based on the sorting results is expressed as follows: ； in This represents the first i samples that have the closest global distance to the query image q; This indicates the index position of the i-th minimum value in the set; This represents the global distance between the query image q and the set of potential positive samples P; The best positive samples Represented as: ； in This indicates that the RANSAC algorithm is used to calculate the query image q and the latent positive samples. Image similarity between them; It is a function for maximizing the value.

5. The scene recognition method based on sample mining strategy and location consistency verification according to claim 4, characterized in that, Step 2.1 involves selecting j absolute negative samples. The method is as follows: Determine the exact negative sample set N of the query image q from the database image R based on the GPS tag of the query image q; and determine the best positive sample of the query image q based on the global distance between the query image q and the database image R. Select j absolute negative samples from the exact negative sample set N in ascending order of global distance. At the same time, make absolute negative samples All satisfy: ； in This indicates the query image q and the best positive sample. The global distance; This represents the query image q and the absolute negative samples. The global distance; This indicates the set threshold.

6. The scene recognition method based on sample mining strategy and location consistency verification according to claim 5, characterized in that, The global distance loss function mentioned in step 2.4 is: ； in Represents the global loss value; The local distance loss function is: ； in This represents the local loss value; This indicates the query image q and the best positive sample. Local distance; This represents the query image q and the absolute negative samples. The local distance.

7. The scene recognition method based on sample mining strategy and location consistency verification according to claim 6, characterized in that, The fusion loss function mentioned in step 2.5 is: ； in This represents the fusion loss value; The weights represent the local distance loss function.

8. The scene recognition method based on sample mining strategy and location consistency verification according to claim 7, characterized in that, The similarity between the query image q and the scene in the database image mentioned in step 3.2 is: ； in Indicates the similarity between images; These represent the global feature descriptors for the queried image and the image in the database, respectively. This represents the modulo operation of a vector.

9. The scene recognition method based on sample mining strategy and location consistency verification according to claim 8, characterized in that, The bidirectional nearest neighbor local feature matching set MNN mentioned in step 3.4 is: ； in This represents the label of the successfully matched local feature in the local features of the candidate frame image after a bidirectional matching operation between the candidate frame image and the query image. ; This represents the label of the successfully matched local feature in the local features of the query image after a bidirectional matching operation between the candidate frame image and the query image. ; Represents any local feature in the candidate frame image; This indicates a specified local feature in the query image; Represents a specified local feature in a candidate frame image; This indicates any local feature in the query image; This represents a similarity measure.

Images