Attribute-aware hashing coding method for large-scale fine-grained image retrieval, storage medium and electronic device

Abstract

This invention discloses an attribute-aware hash coding method for large-scale fine-grained image retrieval, comprising the following steps: extracting multi-scale image features using a convolutional neural network and fusing features at different scales using an upsampling strategy; using a feature mining module consisting of a self-attention mechanism and a feedforward network to mine significantly discriminative features in the image; interacting with the extracted features using a set of learnable queries to directly decouple a set of specific visual attribute features that can distinguish different categories of images; each query-captured specific visual attribute feature is compressed as a one-bit hash code; furthermore, an additional auxiliary branch is used during training to effectively alleviate the optimization difficulties in low-bit hash coding scenarios, enabling better optimization of learnable queries and improving the retrieval performance of hash codes. This invention effectively captures significantly discriminative visual attribute features from complex global image features by setting a set of learnable queries, ensuring that the final generated hash code retains the key semantic information of the image and achieves higher image retrieval accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to an attribute-aware hash coding learning method for large-scale fine-grained image retrieval. Background Technology

[0002] In computer vision and pattern recognition, fine-grained image retrieval aims to retrieve images belonging to multiple different subcategories within the same basic category, such as different species of plants and animals, different car models, and different types of retail products. Its main challenge lies in understanding and distinguishing the subtle visual differences between objects that are very similar in overall appearance but differ in fine-grained features. To address these challenges, exploring and fully utilizing subtle differences at the object attribute level is crucial. Fine-grained image retrieval is an important component of fine-grained image analysis. Compared to coarse-grained image retrieval tasks, it is closer to real-world scenarios and has received increasing attention in recent years. Early work mainly focused on using long real-valued vectors to improve retrieval accuracy. Although these methods have achieved good results, they still face the challenge of excessive time consumption when searching for the nearest neighbors of a given image on large-scale datasets. Hash-based retrieval is another representative strategy for approximate nearest neighbor search. Hash learning, by mapping features to compact binary codes, greatly improves storage efficiency and retrieval speed, becoming a promising solution in fine-grained image retrieval.

[0003] Hash methods are broadly classified into two categories: data-independent methods and data-dependent methods. In data-independent hashing methods, the hash function in the model is typically generated randomly and independently of any training data, but improvements in retrieval performance come at the cost of longer hash codes. Data-dependent hashing methods attempt to learn the hash function from some training data, known as learned hashing algorithms. Compared to data-independent methods, learned hashing algorithms can achieve higher accuracy with shorter hash codes. Therefore, learned hashing algorithms are more popular than data-independent methods in practical applications. With the rise of deep learning, some learned hashing methods have integrated deep feature learning into the hashing framework, achieving excellent performance. In previous work, many deep hashing methods have been proposed for large-scale image retrieval. Compared to deep unsupervised hashing methods, deep supervised hashing methods can fully exploit semantic information and achieve higher retrieval accuracy.

[0004] With the explosive growth of fine-grained data in practical applications, fine-grained hashing has proven to be one of the most promising solutions for handling large-scale fine-grained retrieval tasks, demonstrating significant advantages. By learning compact binary hash codes, fine-grained hashing can greatly reduce storage costs and improve query speed. However, the fine-grained hashing learning process faces the challenge of how to represent rich image details using hash codes of extremely limited length to distinguish different categories of objects with only subtle differences. Some existing deep hashing methods focus on the design of feature extraction modules while neglecting the process of generating hash codes. This results in the final hash codes being derived by simply projecting complexly coupled image features through linear layers. These methods ignore the utilization of the hierarchical information of visual attributes in fine-grained graphics, which enables this invention to effectively identify different categories. Therefore, this invention proposes a novel method to generate attribute-aware hash codes suitable for fine-grained image retrieval. Furthermore, during training, this invention introduces an additional auxiliary branch solely for training, effectively alleviating the optimization difficulties during training in low-bit hashing scenarios and improving model performance. Summary of the Invention

[0005] The purpose of this invention is to provide a method for generating attribute-aware hash codes based on query optimization in large-scale fine-grained image retrieval.

[0006] The technical solution for achieving the objective of this invention is as follows:

[0007] An attribute-aware hashing encoding method for large-scale fine-grained image retrieval includes the following steps:

[0008] Step 1: Extract multi-scale features of the image through a convolutional neural network, fuse the multi-scale features, and use a feature refinement module composed of a multi-head self-attention mechanism and a forward propagation network to mine salient regions in the image to obtain the global features of the image.

[0009] Step 2: Use a set of learnable queries to capture key, distinguishable, specific visual attribute features of the image directly from the extracted global features.

[0010] Step 3 introduces an additional auxiliary branch based on step 2 to alleviate the optimization difficulties in the training process under low bit hashing encoding scenarios, so that learnable queries can capture more discriminative visual attribute features.

[0011] Step 4: Discard the auxiliary branches and use the trained and optimized query to map each specific visual attribute feature obtained into a hash bit.

[0012] Compared with the prior art, the significant advantages of this invention are:

[0013] (1) In fine-grained tasks, images typically contain extremely rich and subtle features. To obtain richer representations, this invention constructs a top-down architecture to capture multi-scale features, enabling the model to effectively capture local features of the image. Simultaneously, a self-attention mechanism is incorporated to eliminate irrelevant information, allowing the model to focus on identifying key regions.

[0014] (2) This invention models the hashing problem as a set prediction problem, where each element in the set represents a one-bit hash code, and this one-bit indicates a specific visual attribute. By constructing the hashing problem in this way, a set of specific attribute features are captured from image features through a set of learnable queries. Each specific attribute feature is then compressed into a final hash code for retrieval.

[0015] (3) This invention introduces additional auxiliary branches to help alleviate the complex optimization challenges often encountered in low-bit hash encoding scenarios and improve model performance. For example... Figure 3 The landscape of the optimization process loss function shows that, compared with higher-dimensional scenarios, the low-bit hash coding scenario ( Figure 3 As shown in (a), the loss function is difficult to reduce to a smaller state. By introducing an auxiliary branch, this invention optimizes in a higher-dimensional space where it is easier to find a smaller loss function state, overcoming the challenges of complex optimization and improving the model's performance.

[0016] (4) The present invention has comparable performance to traditional hash learning methods in coarse-grained image retrieval and far exceeds the retrieval accuracy of traditional hash learning methods in fine-grained image retrieval. Compared with existing fine-grained hash retrieval methods, the present invention also has higher retrieval accuracy and advantages in model complexity and computational overhead. Attached Figure Description

[0017] Figure 1 The flowchart for generating attribute-aware hash codes for this invention is shown.

[0018] Figure 2 This is a schematic diagram of the training process when the number of auxiliary branches is 1.

[0019] Figure 3 A visualization of the complex optimizations encountered in low-bit hash encoding. Detailed Implementation

[0020] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0021] like Figure 1 and Figure 2As shown, this invention provides a method for generating attribute-aware hash codes based on query optimization in large-scale fine-grained image retrieval, specifically including the following steps:

[0022] Step 1: Extract multi-scale features of the image using a convolutional neural network:

[0023]

[0024] Where, Φ CNN (·) represents a convolutional neural network. c represents image features at different scales. j w j and h j Let I represent the number of channels, feature width, and feature length of the depth feature at scale j, respectively, and L represent the number of scales for feature extraction. i Let represent the i-th input image; the multi-scale features are fused as follows:

[0025]

[0026] in, φ represents the multi-scale features after fusion. Conv This indicates a 1×1 convolutional layer used to adjust the number of channels, φ. up This represents the upsampling operation; adjusting the shape of the fused features and stitching together features of different scales yields the extracted feature x. in ∈ d×E Where d represents the adjusted number of feature channels. w j and h j Let represent the feature width and feature length of the depth feature at scale j, respectively. Then, through a multi-head self-attention mechanism and a forward propagation network, the final global features are obtained:

[0027]

[0028]

[0029] Where pos represents the position code. Indicates the position enhancement of x out MHSA(·) represents the multi-head attention mechanism, and FFN(·) represents the forward propagation network. x represents in The transpose of .

[0030] Step 2: Capture image-specific attribute features using learnable queries. Randomly initialize k learnable queries. Each learnable query q iThe global features obtained in step 1 are fed into a decoder for computation.

[0031] Query q i The captured result is the specific attribute feature a output by the decoder. i The specific calculation method is as follows:

[0032]

[0033]

[0034] in, d represents a learnable query. ′ This represents the dimension of each head query, where m = 1, ..., M represents the output from the m-th attention head. Let W represent the learnable projection matrix, softmax(·) represent the normalized exponential function, and Cat(·) represent the concatenation operation. The outputs of each head are concatenated together and processed by W. o Projection yields a i Then, regarding a... i Compression yields:

[0035]

[0036] in, Because the decoding process can be executed in parallel, using Let q represent k learnable queries. Following the process described above, given k q... i It is possible to calculate k h values ​​at once. i Let the entire step 2 be denoted as:

[0037]

[0038] Where Q represents the learnable query and Θ represents the learnable parameters involved in decoding, and Decode(·) indicates that the decoding operation is performed using the decoder, specifically h = [h1; h2; ...; h k ].

[0039] Step 3 is used to overcome Figure 3 This demonstrates the complex optimization problems encountered in low-bit hash encoding. Each of them Divide it evenly into N segments Then, performing a circular shift operation on the sliced ​​subvectors yields:

[0040]

[0041] Where j = 1, ..., N-1. Let... It shares the same set of parameters as the original Q.

[0042] The new gains The result obtained by processing according to the procedure in step 2 is:

[0043]

[0044] This calculation process is performed within an auxiliary branch, which reuses the decoder from step 2.

[0045] Then, the output of the original branch and the output of the auxiliary branch are concatenated together to obtain... SGD is used for optimization based on the loss function, which is denoted as:

[0046]

[0047] in, z i ∈{-1,+1} N×k S is a pair of images where both images belong to the same category. ij =1 otherwise S ij =0. The calculation and optimization of the loss function are performed in an N×k dimensional space.

[0048] Step 4: After the model training is complete, input an image and obtain the result as in Step 1. Ignoring the auxiliary branches, using the parameters trained after step 3, the hash code corresponding to the image can be generated:

[0049]

[0050] Here, sign(·) represents the sign function.

[0051] This invention effectively captures significant and distinguishable visual attribute features from complex global image features by setting a set of learnable queries, so that the final generated hash code retains the visual attribute information of the image and achieves higher image retrieval accuracy.

Claims

1. An attribute-aware hash coding method for large-scale fine-grained image retrieval, characterized in that, Includes the following steps: Step 1: Extract multi-scale features of the image through a convolutional neural network, fuse the multi-scale features, and use a feature refinement module composed of a multi-head self-attention mechanism and a forward propagation network to mine salient regions in the image to obtain the global features of the image. Step 2: Use a set of learnable queries to directly capture key, distinguishable, specific visual attribute features of the image from the extracted global features; Step 3: Based on step 2, an additional auxiliary branch is introduced to alleviate the optimization difficulties in the training process under low bit hashing encoding scenarios, so that the learnable query can capture more discriminative visual attribute features. Step 4: Discard the auxiliary branches and use the trained and optimized query to map each specific visual attribute feature obtained into a hash bit. Step 2 specifically involves: capturing image-specific attribute features using learnable queries; and random initialization. A learnable query Each learnable query The global features obtained in step 1 are fed into a decoder for computation; Query The captured result is the specific attribute feature output by the decoder. The specific calculation method is as follows: ； in, This represents a learnable query. This indicates the dimension of each header query. Indicates from the The output of each attention head, Represents the learnable projection matrix. Represents the normalized exponential function, This indicates a concatenation operation; it concatenates the outputs of each header together. Projection Then on After compression, the result is: ； in, Because the decoding process can be executed in parallel, using express A learnable query, given the above process. indivual It can be calculated in one go. indivual Let the entire step 2 be denoted as: ； in, Indicates the enhanced position , This indicates a query that can be learned. This represents the learnable parameters involved in the decoding process. This indicates that a decoder is used to perform a decoding operation, specifically... ; Step 3 specifically involves: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Each of them Divide it evenly into N segments. Then, a circular shift operation is performed on the sliced ​​subvectors to obtain: ； in, remember , and the original Sharing the same set of parameters; The new gains The result obtained by processing according to the procedure in step 2 is: ； This calculation process is performed within an auxiliary branch, and the auxiliary branch reuses the decoder from step 2; Then, the output of the original branch and the output of the auxiliary branch are concatenated together to obtain... Optimize using SGD based on the loss function, denoted as: ； in, , When both images in an image pair belong to the same category otherwise The calculation and optimization of the loss function are in Execute in 3D space.

2. The attribute-aware hash coding method for large-scale fine-grained image retrieval according to claim 1, characterized in that, Step 1 is as follows: Multi-scale features of images are extracted using convolutional neural networks: ； in, Represents a convolutional neural network. Representing image features at different scales. They represent the first The number of channels, feature width, and feature length of scale-depth features. Indicates the number of scales for feature extraction. Indicates the first Input image; fuse multi-scale features as follows: ； in, This represents the multi-scale features after fusion. Represents a layer Convolutional layers are used to adjust the number of channels. This represents the upsampling operation; adjusting the shape of the fused features and stitching together features of different scales to obtain the extracted features. ,in, This indicates the adjusted number of feature channels. , They represent the first The feature width and feature length of the scale-depth features are then further mined using a multi-head self-attention mechanism and a forward propagation network to obtain the final global features. ； ； in, Indicates position code, Indicates the enhanced position , This indicates a multi-head self-attention mechanism. Indicates a forward propagation network. express Transpose of; Step 4 specifically involves: After the model training is complete, input an image and obtain the result as in Step 1. Ignoring auxiliary branches, using the parameters trained after step 3, generate the hash code corresponding to the image: ； in, Represents a symbolic function.

3. A computer storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 2.

4. An electronic device, characterized in that, The device includes a memory and one or more processors, the memory being used to store one or more programs; when the one or more programs are executed by the one or more processors, they implement the method as described in any one of claims 1 to 2.

Images