Attribute perceptual hash coding method for large-scale fine-grained image retrieval, storage medium and electronic equipment

Abstract

The invention discloses an attribute perceptual hash coding method for large-scale fine-grained image retrieval, which comprises the following steps of: extracting multi-scale image features through a convolutional neural network, and fusing the features of different scales by adopting an up-sampling strategy. A feature mining module composed of a self-attention mechanism and a forward network is used for mining features with remarkable discrimination in an image. For the extracted features, a group of learnable queries are used for interacting with the features, and a group of specific visual attribute features capable of distinguishing different types of images are directly decoupled. Specific visual attribute features captured by each query are compressed as one bit of hash coding. Besides, additional auxiliary branches are adopted in the training process, so that the training process can effectively relieve the problem of difficulty in optimization in a low-bit hash coding scene, learnable query can be better optimized, and the retrieval performance of hash coding is improved. According to the method, a group of learnable queries are set to effectively capture remarkably distinguishable visual attribute features from complex global image features, so that the finally generated Hash codes reserve key semantic information of the image, and higher image retrieval accuracy is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an attribute-aware hash coding learning method in large-scale fine-grained image retrieval. Background Art

[0002] In computer vision and pattern recognition, the goal of fine-grained image retrieval is to retrieve images of multiple different subcategories belonging to the same basic category, such as different species of plants and animals, different car models, different kinds of retail products, etc. The main challenge is to understand and distinguish the fine-grained visual differences of objects that are very similar in overall appearance but different in fine-grained features. To address these challenges, it is crucial to explore and make full use of the subtle differences at the object attribute level. Fine-grained image retrieval is an important part of fine-grained image analysis. Compared with coarse-grained image retrieval tasks, it is closer to real scenes and has received increasing attention in recent years. Early works mainly focused on using long real-valued vectors to improve retrieval accuracy. Although these methods have achieved good results, they still face the challenge of excessive time consumption when searching for the nearest neighbors of a given image in large-scale datasets. Hash-based retrieval is another representative strategy for approximate nearest neighbor search. Hash learning greatly improves storage efficiency and retrieval speed by mapping features into compact binary codes, becoming a promising solution in fine-grained image retrieval.

[0003] Hashing methods can be roughly divided into two categories, namely data-independent methods and data-dependent methods. In data-independent hashing methods, the hash function in the model is usually randomly generated and independent of any training data, but the improvement in retrieval performance requires the length of the hash code. Data-dependent hashing methods attempt to learn hash functions from some training data, which is called learned hashing algorithms. Compared with data-independent methods, learned hashing algorithms can achieve higher accuracy with shorter hash codes. Therefore, learned hashing algorithms are more popular than data-independent methods in practical applications. With the rise of deep learning, some learned hashing methods integrate deep feature learning into the hashing framework and achieve good performance. In previous works, many deep hashing methods have been proposed for large-scale image retrieval. Compared with deep unsupervised hashing methods, deep supervised hashing methods can fully mine semantic information and obtain higher retrieval accuracy.

[0004] With the explosive growth of fine-grained data in practical applications, fine-grained hashing has been proven to have significant advantages as one of the most promising solutions for handling large-scale fine-grained retrieval tasks. By learning compact binary hash codes, fine-grained hashing can greatly reduce storage costs and improve query speed. However, the fine-grained hashing learning process faces the problem of how to use hash codes with extremely limited length to represent rich image details to distinguish objects of different categories with only slight differences. Some existing deep hashing methods focus on the design of feature extraction modules and ignore the process of generating hash codes. The resulting hash codes are obtained by simply projecting complex coupled image features through linear layers. These methods ignore the use of visual attribute hierarchical information of fine-grained graphics, which enables the present invention to effectively identify different categories. Therefore, the present invention proposes a novel method to generate attribute-aware hash codes suitable for fine-grained image retrieval. In addition, during the training process, the present invention introduces additional auxiliary branches only for training, which effectively alleviates the optimization difficulties in the training process in low-bit hash coding scenarios and improves model performance. Summary of the invention

[0005] The purpose of the present invention is to provide a method for generating attribute-aware hash codes based on query optimization in large-scale fine-grained image retrieval.

[0006] The technical solution for achieving the purpose of the present invention is:

[0007] The attribute-aware hash coding method for large-scale fine-grained image retrieval includes the following steps:

[0008] Step 1: extract the multi-scale features of the image through a convolutional neural network, fuse the multi-scale features, and use a feature refinement module consisting of a multi-head self-attention mechanism and a forward propagation network to mine the salient areas in the image to obtain the global features of the image;

[0009] In step 2, a set of learnable queries are used to capture the key distinguishable specific visual attribute features of the image directly from the extracted global features.

[0010] Step 3: Based on step 2, additional auxiliary branches are introduced to alleviate the optimization difficulties in the training process in the low-bit hash coding scenario, so that the learnable queries can capture more discriminative visual attribute features.

[0011] Step 4: discard the auxiliary branch and use the trained and optimized query to map each specific visual attribute feature obtained into a hash bit.

[0012] Compared with the prior art, the present invention has the following significant advantages:

[0013] (1) In fine-grained tasks, images usually contain extremely rich and subtle features. In order to obtain a richer representation, the present invention constructs a top-down architecture to capture multi-scale features, so that the model can effectively capture the local features of the image. At the same time, a self-attention mechanism is incorporated to eliminate irrelevant information, so that the model can focus on identifying key areas.

[0014] (2) The present invention models the hash problem as a set prediction problem, where each element in the set represents a bit of the hash code, and this bit can indicate a specific visual attribute. By constructing the hash problem in this way, a set of specific attribute features are captured from the image features through a set of learnable queries. Then, each specific attribute feature is compressed into a final hash code for retrieval.

[0015] (3) The present invention introduces additional auxiliary branches to help alleviate the complex optimization challenges often encountered in low-bit hash coding scenarios and improve the performance of the model. Figure 3 The loss function landscape of the optimization process is shown in Figure 2. Compared with the higher dimensional scenario, the low-bit hash coding scenario ( Figure 3 a) The loss function is difficult to drop to a smaller state. By introducing auxiliary branches, the present invention optimizes in a higher-dimensional space where it is easier to find a smaller loss function state, overcomes the challenges of complex optimization and improves the performance of the model.

[0016] (4) The present invention has performance results comparable to those of traditional hash learning methods in coarse-grained image retrieval, and has a retrieval accuracy far exceeding that of traditional hash learning methods in fine-grained image retrieval; compared with existing fine-grained hash retrieval methods, the present invention also has a higher retrieval accuracy, and the present invention also has advantages in model complexity and computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flowchart for generating attribute-aware hash codes for the present invention.

[0018] Figure 2 Schematic diagram of the training process when the number of auxiliary branches is 1.

[0019] Figure 3 Visualization of the optimization complexity encountered for low-bit hash encoding. DETAILED DESCRIPTION

[0020] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0021] like Figure 1 and Figure 2As shown, the present invention provides a method for generating attribute-aware hash codes based on query optimization in large-scale fine-grained image retrieval, which specifically includes the following steps:

[0022] Step 1: Extract the multi-scale features of the image through a convolutional neural network:

[0023]

[0024] Among them, Φ CNN (·) represents a convolutional neural network, Represents image features of different scales, c j 、w j and h j They represent the number of channels, feature width and feature length of the j-th scale depth feature, L represents the number of scales of extracted features, and I i Represents the i-th input image; the multi-scale features are fused as follows:

[0025]

[0026] in, represents the fused multi-scale features, φ Conv Represents a 1×1 convolutional layer used to adjust the number of channels, φ up Represents the upsampling operation; adjusting the shape of the fused features and splicing features of different scales together can obtain the proposed feature x in ∈ d×E , where d represents the number of feature channels after adjustment, w j and h j They represent the feature width and feature length of the j-th scale depth feature respectively. The final global feature is obtained by further mining through the multi-head self-attention mechanism and the forward propagation network:

[0027]

[0028]

[0029] Among them, pos represents the position code, Represents x after position enhancement out , MHSA(·) represents the multi-head attention mechanism, FFN(·) represents the forward propagation network, Represents x in The transpose of .

[0030] Step 2: Use learnable queries to capture image-specific attribute features. Randomly initialize k learnable queries Each learnable query q iThe global features obtained in step 1 are fed into a decoder for calculation.

[0031] Query q i The captured result is the specific attribute feature a output by the decoder i , and its specific operation method is as follows:

[0032]

[0033]

[0034] in, represents a learnable query, d ′ is the dimension of each head query, m=1,…,M represents the output from the mth attention head, represents the learnable projection matrix, softmax(·) represents the normalized exponential function, and Cat(·) represents the concatenation operation. The output of each head is concatenated together and passed through W o Projection gets a i . Then do a i Compression yields:

[0035]

[0036] in, Since the decoding process can be performed in parallel, using Represents k learnable queries. Following the above process, given k q i , k h can be calculated at one time i . The entire step 2 is recorded as:

[0037]

[0038] Where Q represents the learnable query and Θ represents the learnable parameters involved in decoding. Decode(·) represents the decoding operation performed using a decoder, specifically h = [h1; h2; ...; h k ].

[0039] Step 3, to overcome Figure 3 The low-bit hash encoding presented here presents a complex optimization problem. Each of Divide it into N segments evenly Then perform a circular shift operation on the sliced ​​subvectors to obtain:

[0040]

[0041] Where j = 1, ..., N-1. Shares the same set of parameters as the original Q.

[0042] The new Follow the process in step 2 to get:

[0043]

[0044] This computation is performed in the auxiliary branch, which reuses the decoder from step 2.

[0045] After that, the output of the original branch and the output of the auxiliary branch are spliced ​​together to obtain SGD is used for optimization according to the loss function, and the loss function is recorded as:

[0046]

[0047] in, z i ∈{-1,+1} N×k .When the two images of an image pair belong to the same category S ij =1 otherwise S ij =0. The calculation and optimization of the loss function are performed in N×k dimensional space.

[0048] Step 4: After the model training is completed, input an image and get the result according to step 1. Ignoring the auxiliary branch, using the trained parameters after executing step 3, we can generate the hash code corresponding to the image:

[0049]

[0050] Wherein, sign(·) represents the sign function.

[0051] The present invention effectively captures significantly distinguishable visual attribute features from complex global image features by setting a set of learnable queries, so that the hash code finally generated retains the visual attribute information of the image and achieves a higher image retrieval accuracy.

Claims

1. An attribute-aware hash coding method for large-scale fine-grained image retrieval, characterized in that: The following steps are involved: Step 1: extract the multi-scale features of the image through a convolutional neural network, fuse the multi-scale features, and use a feature refinement module consisting of a multi-head self-attention mechanism and a forward propagation network to mine the salient areas in the image to obtain the global features of the image; Step 2: Use a set of learnable queries to capture the key distinguishable specific visual attribute features of the image directly from the extracted global features; Step 3: Based on step 2, an additional auxiliary branch is introduced to alleviate the optimization difficulty in the training process in the low-bit hash coding scenario, so that the learnable query can capture more discriminative visual attribute features; Step 4: discard the auxiliary branch and use the trained and optimized query to map each specific visual attribute feature obtained into a hash bit.

2. The attribute-aware hash coding method for large-scale fine-grained image retrieval according to claim 1, characterized in that: Step 1 is as follows: The multi-scale features of the image are extracted through the convolutional neural network: Among them, Φ CNN (·) represents a convolutional neural network. Represents image features of different scales, c j 、w j and h j They represent the number of channels, feature width and feature length of the j-th scale depth feature, L represents the number of scales of extracted features, and I i Represents the i-th input image; the multi-scale features are fused as follows: in, represents the fused multi-scale features, φ Conv Represents a 1×1 convolutional layer used to adjust the number of channels, φ up Represents the upsampling operation; adjusts the shape of the fused features and concatenates features of different scales together to obtain the proposed feature x in ∈ d×E , where d represents the number of feature channels after adjustment, w j and h j They represent the feature width and feature length of the j-th scale depth feature respectively; and then further mined through the multi-head self-attention mechanism and the forward propagation network to obtain the final global feature: Among them, pos represents the position code, Represents x after position enhancement out , MHSA(·) represents the multi-head self-attention mechanism, FFN(·) represents the forward propagation network, Represents x in The transpose of Step 2: Use learnable queries to capture image-specific attribute features; randomly initialize k learnable queries Each learnable query q i The global features obtained in step 1 are fed into a decoder for calculation; Query q i The captured result is the specific attribute feature a output by the decoder i , and its specific operation method is as follows: in, represents a learnable query, d ′ represents the dimension of each head query, m=1,…,M represents the output from the mth attention head, represents the learnable projection matrix, softmax(·) represents the normalized exponential function, and Cat(·) represents the concatenation operation; the output of each head is concatenated together and passed through W o Projection gets a i ; Then do a i Compress it to get: in, Since the decoding process can be performed in parallel, using Represents k learnable queries. Following the above process, given k q i , k h can be calculated at one time i ; Record the entire step 2 as: Where Q represents a learnable query, Θ represents the learnable parameters involved in decoding, and Decode(·) represents the decoding operation performed using a decoder. Specifically, h=[h1;h2;…;h k ]; Step 3, Each of Divide it into N segments evenly Then perform a circular shift operation on the sliced ​​subvector to obtain: Where j = 1, ..., N-1. Shares the same set of parameters as the original Q; The new Follow the process in step 2 to get: This computation is performed in the auxiliary branch, which reuses the decoder from step 2. After that, the output of the original branch and the output of the auxiliary branch are spliced ​​together to obtain SGD is used for optimization according to the loss function, and the loss function is recorded as: in, z i ∈{-1,+1} N×k .When the two images of an image pair belong to the same category S ij =1 otherwise S ij = 0. The calculation and optimization of the loss function are performed in N×k dimensional space; Step 4: After the model training is completed, input an image and get the result according to step 1. Ignore the auxiliary branch and use the trained parameters after executing step 3 to generate the hash code corresponding to the image: Wherein, sign(·) represents the sign function.

3. A computer storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

4. An electronic device, characterized in that: It comprises a memory and one or more processors, wherein the memory is used to store one or more programs; when the one or more programs are executed by the one or more processors, the method according to any one of claims 1 to 2 is implemented.

Images