A two-stage place recognition method based on similarity constraint deep hashing

Abstract

A two-stage location identification method based on similarity-constrained deep hashing includes: First, in the first stage, compact binary hash features are used for fast retrieval using similarity-constrained deep hashing, significantly reducing memory usage and improving retrieval efficiency. Next, in the second stage, higher-dimensional floating-point features are used for precise re-ranking, ensuring the accuracy of the identification results. Furthermore, by combining similarity-constrained quantization loss and a straight-through estimator (STE), end-to-end training with quantization operations is achieved, avoiding parameter update problems caused by zero gradients. Validation on multiple datasets shows that this invention significantly reduces retrieval latency and storage space while significantly improving Rank-1 accuracy and mean accuracy (mAP), providing an efficient and accurate solution for applications in mobile robot localization and augmented reality, and possessing significant practical application value.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, and in particular to a two-stage location recognition method based on similarity-constrained deep hashing. Background Technology

[0002] Visual place recognition (VPR), also known as image localization or visual geolocation, aims to roughly estimate the location of a query location image by searching for the best match from a geotagged image database. VPR has wide applications in fields such as mobile robot localization and augmented reality.

[0003] Image profiling (VPR) tasks typically use global or local descriptors to represent image features. Aggregation algorithms such as VLAD are often used to aggregate local features into a vector as global features. This compact global feature set facilitates fast location retrieval and is robust to viewpoint changes. However, these global features ignore spatial information, making VPR methods based on them prone to perceptual aliasing. Current solutions employ two-stage VPR: using global features to retrieve the top k results from the database as candidates, and then performing spatial validation by reordering these candidates by matching local features. VPR model training follows a "train first, fine-tune later" approach. Most VPR models are initialized using model parameters pre-trained on the ImageNet dataset and fine-tuned on VPR datasets such as Mapillary street-level sequences (MSLS). As models and training datasets continue to expand, two-stage training becomes increasingly expensive in terms of computation and memory usage.

[0004] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a two-stage location identification method based on similarity-constrained deep hashing.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A two-stage location identification method based on similarity-constrained deep hashing includes the following steps:

[0008] S1. Based on the triplet training framework, construct triplet training samples containing query image, positive example image and negative example image; input the training samples into the feature extraction network to extract image features for two-stage retrieval;

[0009] S2. Perform dimensionality reduction and dimensionality increase processing on the high-dimensional semantic feature map output by the feature extraction network; convert the dimensionality-reduced features into binary hash features using a similarity-constrained deep hashing method, and simultaneously obtain the dimensionality-incremented floating-point features;

[0010] S3. Use binary hash features to perform a fast first-stage retrieval in the image database to filter out the top k candidate images that are similar to the query image features; use floating-point features to perform a precise reordering of the candidate images in the second stage to determine the final top k images that are most similar to the query image.

[0011] S4. Using a weakly supervised method, calculate the overall loss function that includes quantization loss and measurement loss with similarity constraints; iteratively optimize the model parameters until the preset convergence condition is met to achieve end-to-end training of the model.

[0012] Furthermore, in step S1, the construction of the triplet training samples is based on geographic location labels, with each query image corresponding to several positive and negative example images, and the model training is optimized through an online hard mining strategy.

[0013] Furthermore, in step S1, the feature extraction network uses a pre-trained large visual model to extract high-dimensional semantic feature maps, and aggregates the high-dimensional semantic feature maps into one-dimensional feature vectors through generalized mean pooling (Gempooling).

[0014] Furthermore, in step S2, the high-dimensional semantic feature map is subjected to dimensionality reduction and dimensionality increase processing through linear layers and fully connected layers, and then binary hash features and floating-point features are generated respectively.

[0015] Furthermore, in step S2, the similarity-constrained deep hashing method constrains the similarity between binary hash features and floating-point features by quantizing the loss constraint, and combines it with a straight-through estimator (STE) to achieve end-to-end training.

[0016] Furthermore, in step S3, the first stage of retrieval uses fast matching of binary hash features to filter out the top k candidate images that are most similar to the query image features, and the second stage of reordering uses the similarity measure between floating-point features to determine the final top k images that are most similar to the query image.

[0017] Furthermore, in step S4, the overall loss function combines quantization loss and metric loss through weighted summation, and iteratively optimizes the model parameters using the backpropagation algorithm. The convergence condition for model optimization is that the error value calculated by the loss function is less than a preset threshold, or the number of iterations reaches a predetermined value. This weakly supervised method improves the model's retrieval accuracy and efficiency through the joint optimization of metric loss and quantization loss.

[0018] Furthermore, in step S4, the measurement loss includes multiple similarity loss (MS loss), which is used to optimize the similarity measurement between image features.

[0019] Further, in step S4, the measurement loss of the similarity constraint is calculated through the following steps:

[0020] Calculate the quantization loss: Based on the similarity difference between floating-point features and binary hash features, calculate the quantization loss;

[0021] Calculate the metric loss: Calculate the metric loss based on the multi-similarity loss, and use a pass-through estimator to calculate the gradient during backpropagation;

[0022] Calculate the overall loss: The overall loss function is obtained by combining the quantization loss and the measurement loss through a weighted summation.

[0023] A computer program product includes a computer program that, when executed by a processor, implements the two-stage location identification method based on similarity-constrained deep hashing.

[0024] The present invention has the following beneficial effects:

[0025] This invention provides an efficient two-stage visual location recognition method, innovatively solving the key retrieval latency and storage problems in visual location recognition applications, and significantly reducing the model's consumption in these two aspects. First, by introducing a similarity-constrained deep hashing two-stage location recognition paradigm, image features are obtained through hashing for the first-stage retrieval, and then reordered in the second stage using higher-dimensional floating-point features. Using compact binary features in the first-stage retrieval significantly reduces VPR memory consumption and improves the model's retrieval efficiency. This invention proposes a similarity-constrained quantization loss, which effectively resists the quantization loss and metric loss resistance problems existing in deep hashing training. Furthermore, it preferably integrates binary hashing operations into the neural network by combining a straight-through estimator (STE), achieving end-to-end training with quantization operations. Quantization is performed during forward propagation, avoiding the situation where parameters cannot be updated when the gradient is zero during backpropagation. Compared to traditional two-stage visual location recognition methods, the method of this invention significantly reduces retrieval latency and storage space on multiple datasets, including Pittsburgh, Tokyo24 / 7, and Mapillary Street-Level Sequences (MSLS), while improving Rank-1 accuracy and mean AP. This invention can directly match local features without spatial verification, making re-ranking much faster than mainstream two-stage VPR methods and requiring less memory. This invention is applicable to visual location recognition scenarios and has significant application value in mobile robot localization and augmented reality fields.

[0026] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0027] Figure 1 This is a flowchart of a two-stage location identification method based on similarity-constrained deep hashing according to an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of a two-stage location identification method based on similarity-constrained deep hashing, according to an embodiment of the present invention. Detailed Implementation

[0029] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0030] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0031] See Figure 1 and Figure 2 This invention provides a two-stage location identification method based on similarity-constrained deep hashing, comprising the following steps:

[0032] Step S1, Data Preparation and Feature Extraction: Based on the triplet training framework, construct triplet training samples containing query image, positive example image and negative example image; input the training samples into the feature extraction network to extract image features for two-stage retrieval.

[0033] The triplet training framework can be, for example, the training framework for the GSV-Cities dataset.

[0034] In a preferred embodiment, in step S1, the construction of the triplet training samples is based on geographic location labels, each query image corresponds to several positive and negative example images, and the model training is optimized through an online hard mining strategy.

[0035] In a preferred embodiment, the feature extraction network uses a pre-trained large visual model (e.g., the DINOv2 model) to extract high-dimensional semantic feature maps, and aggregates the high-dimensional semantic feature maps into one-dimensional feature vectors through generalized mean pooling (Gempooling).

[0036] Step S2, Feature Processing and Transformation: The high-dimensional semantic feature map output by the feature extraction network is subjected to dimensionality reduction and dimensionality increase processing; the dimensionality-reduced features are converted into binary hash features through a similarity-constrained deep hashing method, while the dimensionality-incremented floating-point features are obtained.

[0037] In a preferred embodiment, in step S2, the high-dimensional semantic feature map is subjected to dimensionality reduction and dimensionality increase processing through linear layers and fully connected layers, and then binary hash features and floating-point features are generated respectively.

[0038] In a preferred embodiment, the similarity-constrained deep hashing method constrains the similarity between binary hash features and floating-point features by quantizing the loss constraint, and combines it with a straight-through estimator (STE) to achieve end-to-end training.

[0039] Step S3, Two-stage retrieval: First-stage fast retrieval is performed in the image database using binary hash features to filter out the top k candidate images with features similar to the query image; second-stage precise reordering of the candidate images is performed using floating-point features to determine the final top k images most similar to the query image.

[0040] Specifically, the first stage of retrieval uses fast matching of binary hash features to filter out the top k candidate images that are most similar to the query image features. The second stage of reordering uses the similarity measure between floating-point features to determine the final top k images that are most similar to the query image.

[0041] Step S4, Model Optimization: Using a weakly supervised method, calculate the overall loss function that includes quantization loss and metric loss, which contain similarity constraints; iteratively optimize the model parameters until the preset convergence condition is met, thus achieving end-to-end training of the model.

[0042] In a preferred embodiment, in step S4, the overall loss function combines quantization loss and metric loss through weighted summation, and iteratively optimizes the model parameters using a backpropagation algorithm. The convergence condition for model optimization is that the error value calculated by the loss function is less than a preset threshold, or the number of iterations reaches a predetermined value. This weakly supervised method improves the model's retrieval accuracy and efficiency through the joint optimization of metric loss and quantization loss.

[0043] In a preferred embodiment, the metric loss includes multiple similarity loss (MS loss) to optimize the similarity metric between image features.

[0044] In a further preferred embodiment, the metric loss of the similarity constraint is calculated through the following steps: quantization loss is calculated based on the similarity difference between floating-point features and binary hash features; metric loss is calculated based on multi-similarity loss, and gradient is calculated during backpropagation using a pass-through estimator; the overall loss function is obtained by combining quantization loss and metric loss through weighted summation.

[0045] This invention proposes a highly efficient two-stage visual location recognition method, effectively addressing the shortcomings of traditional methods in terms of retrieval latency and storage consumption. In the first stage, compact binary hash features are used for rapid retrieval, significantly reducing memory usage and improving retrieval efficiency. The second stage utilizes higher-dimensional floating-point features for precise re-sorting, ensuring the accuracy of the recognition results. Furthermore, by combining similarity-constrained quantization loss with a straight-through estimator (STE), end-to-end training with quantization operations is achieved, avoiding parameter update problems caused by zero gradients and further improving the model's training performance. Compared with existing technologies, this invention has demonstrated its superiority on multiple datasets, significantly reducing retrieval latency and storage space while also significantly improving Rank-1 accuracy and mean accuracy (mAP). It provides a more efficient and accurate solution for applications in mobile robot localization and augmented reality, possessing significant practical application value.

[0046] The following further describes specific embodiments of the present invention and examples of its algorithms.

[0047] An efficient two-stage visual location recognition and model training method is proposed, which solves the inefficiency of memory usage during the training process of traditional two-stage visual location recognition methods, and reduces the retrieval latency and expensive storage costs during the second-stage reordering. The method includes: in the first stage, deep hashing is used to generate low-dimensional binary global hash features to select the top K candidate images most similar to the query image; then, higher-dimensional floating-point features are used to reorder the K candidate images; a similarity-constrained deep hashing method is adopted, replacing the traditional deep hashing quantization loss L1 / L2 with the similarity-constrained quantization loss L_Q, and combining straight-through estimation (STE) to integrate the binary hashing operation into the neural network, achieving end-to-end training of the network containing quantization operations.

[0048] See Figure 1 and Figure 2 A two-stage location identification method based on similarity-constrained deep hashing includes the following steps:

[0049] Step S1: Using the training framework of the GSV-Cities dataset, each query image in a batch corresponds to several positive and negative images to form a triplet for training. The mini-batch of training image triplets with geographic location labels is input into the feature extraction network to extract image features for two-stage retrieval.

[0050] Step S2: The high-dimensional semantic feature map obtained by the feature extraction network is sequentially passed through a linear layer, an aggregation layer, and a fully connected layer for dimensionality reduction and dimensionality increase, respectively. The dimensionality-reduced features are subjected to similarity-constrained deep hashing to obtain binary hash features, while the dimensionality-increased features are obtained as floating-point features.

[0051] Step S3: First, use binary hash features to perform a first-stage retrieval of the query image in the image database to obtain the top k candidate images that are most similar to the query image features. Then, use floating-point features to perform a more precise reordering of the candidate k images to obtain the final top k images that are most similar to the query image.

[0052] Step S4: Using a weakly supervised method, calculate the overall loss function, including the measurement loss and quantization loss of similarity constraints, and iteratively optimize the model parameters until the training meets the preset convergence conditions.

[0053] Specifically, in a preferred embodiment, the training method for the two-stage visual position recognition further includes the following steps:

[0054] Further, in step S2, the feature extraction network adopts the pre-trained visual large model DINOv2 model, uses the patch token output by DINO v2 as a high-dimensional feature map, and then uses generalized mean pooling (Gem pooling) to aggregate it into a one-dimensional feature vector in the aggregation layer.

[0055] Furthermore, in step S4, the measurement loss of the similarity constraint is specifically calculated through the following steps:

[0056] S41. Calculate the quantization loss of similarity constraints:

[0057]

[0058] Among them, f i f j Let b be the floating-point features of images i and j. i b j Let K be the hash features of images i and j, and K represent the number of pairs of images in a batch.

[0059] S42. Calculate the metric loss L calculated using the multiple similarity loss (MS loss). M (b) and a straight-through estimation (STE) is used to facilitate gradient calculation during backpropagation.

[0060] S43. Calculate the total loss L = L M (b)+λL Q(f,b), where λ controls the weights of the quantization loss and the measurement loss.

[0061] During training, training data is continuously input into the model from steps S1 to S4 to calculate the overall loss function and iteratively optimize the model parameters until the training meets the preset convergence condition. Specifically, the model error is calculated using a metric loss function and a quantization loss function, and the model parameters are updated based on the error using a backpropagation algorithm to optimize the performance of the linear layer, aggregation layer, and fully connected layer. The training result meeting the preset convergence condition can be achieved when the error value calculated by the loss function is less than a preset threshold, or when the number of iterations reaches a predetermined value.

[0062] Compared to traditional two-stage visual location recognition methods, the method of this invention significantly reduces retrieval latency and storage space on multiple datasets, including Pittsburgh, Tokyo24 / 7, and Mapillary Street-Level Sequences (MSLS), while improving Rank-1 accuracy and mean AP. This invention can directly match local features without spatial verification, making re-ranking much faster than mainstream two-stage VPR methods and requiring less memory.

[0063] In summary, the two-stage visual position recognition method of this invention significantly reduces retrieval latency and storage space compared to traditional methods, while improving Rank-1 accuracy and mean accuracy (mAP). This invention is applicable to visual position recognition scenarios and has significant application value in the fields of mobile robot localization and augmented reality.

[0064] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0065] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0066] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0067] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk drive or magnetic tape drive. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0068] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0069] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0070] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0071] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0072] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0073] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0074] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0075] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0076] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A two-stage location identification method based on similarity-constrained deep hashing, characterized in that, Includes the following steps: S1. Based on the triplet training framework, construct triplet training samples containing query image, positive example image and negative example image; input the training samples into the feature extraction network to extract image features for two-stage retrieval; S2. Perform dimensionality reduction and dimensionality increase processing on the high-dimensional semantic feature map output by the feature extraction network; convert the dimensionality-reduced features into binary hash features using a similarity-constrained deep hashing method, and simultaneously obtain the dimensionality-incremented floating-point features; S3. Use binary hash features to perform a fast first-stage retrieval in the image database to filter out the top k candidate images that are similar to the query image features; use floating-point features to perform a precise reordering of the candidate images in the second stage to determine the final top k images that are most similar to the query image. S4. Using a weakly supervised method, calculate the overall loss function including the quantization loss and the measurement loss of similarity constraints; iteratively optimize the model parameters until the preset convergence condition is met to achieve end-to-end training of the model; the measurement loss of the similarity constraints is calculated through the following steps: Calculate the quantization loss: Based on the similarity difference between floating-point features and binary hash features, calculate the quantization loss; Calculate the metric loss: Calculate the metric loss based on the multi-similarity loss, and use a pass-through estimator to calculate the gradient during backpropagation; Calculate the overall loss: The overall loss function is obtained by combining the quantization loss and the measurement loss through a weighted summation.

2. The two-stage location identification method as described in claim 1, characterized in that, In step S1, the construction of the triplet training samples is based on geographic location labels. Each query image corresponds to several positive and negative example images, and the model training is optimized through an online hard mining strategy.

3. The two-stage location identification method as described in claim 1 or 2, characterized in that, In step S1, the feature extraction network uses a pre-trained large visual model to extract high-dimensional semantic feature maps, and aggregates the high-dimensional semantic feature maps into one-dimensional feature vectors through generalized mean pooling (Gempooling).

4. The two-stage location identification method as described in any one of claims 1 to 2, characterized in that, In step S2, the high-dimensional semantic feature map is processed by linear layer and fully connected layer for dimensionality reduction and dimensionality increase, and then binary hash feature and floating-point feature are generated respectively.

5. The two-stage location identification method as described in any one of claims 1 to 2, characterized in that, In step S2, the similarity-constrained deep hashing method constrains the similarity between binary hash features and floating-point features by quantizing the loss, and combines it with a straight-through estimator (STE) to achieve end-to-end training.

6. The two-stage location identification method as described in any one of claims 1 to 2, characterized in that, In step S3, the first stage of fast retrieval uses binary hash features to quickly match and select the top k candidate images that are most similar to the query image features. The second stage of precise reordering uses the similarity measure between floating-point features to determine the final top k images that are most similar to the query image.

7. The two-stage location identification method as described in any one of claims 1 to 2, characterized in that, In step S4, the model parameters are iteratively optimized using the backpropagation algorithm. The convergence condition for model optimization is that the error value calculated by the loss function is less than a preset threshold, or the number of iterations reaches a predetermined value. The weakly supervised method improves the retrieval accuracy and efficiency of the model through joint optimization of metric loss and quantization loss.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the two-stage location identification method as described in any one of claims 1 to 7.

Images