A deep hashing retrieval method based on classification labels

Abstract

The present invention discloses a deep hash retrieval method based on classification labels. First, a deep convolutional neural network is used to extract features from images in a training image database. A deep hash network is constructed after the main network to learn class hash codes, and a classifier is constructed after the deep hash network to learn classification features. Secondly, the class hash codes and classification features are used to calculate the metric learning loss and classification loss respectively, and the class hash codes and hash codes are used to calculate the quantization loss. Finally, in the test phase, the classification labels are binarized and concatenated with the hash codes, and the resulting hash codes based on the classification labels are used for retrieval. The method proposed in the present invention utilizes both classification information and similarity information, and can achieve good retrieval accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image retrieval and relates to a deep hash retrieval method based on classification labels. Background Art

[0002] The general process of deep hash retrieval is to use a deep hash network to process high-dimensional features to obtain low-dimensional real-valued features, and then obtain binary hash codes through quantization operations. The network learning ability can be enhanced through methods such as metric learning and quantization loss, thereby improving the hash retrieval effect.

[0003] However, the hash code obtained by such an image retrieval method based on image pair similarity measurement can only save the distribution information of the image in the original space, the visual content information of the image itself, and the retrieval result of a given query image returned from the database. It cannot further meet the needs of image analysis and processing. For current image retrieval methods, for a given query image, while returning a set of similar images, their semantic label information cannot be obtained.

[0004] By fusing the similarity information between image pairs with the semantic information of each image, we can generate more discriminative feature representations, leading to better retrieval and classification results. Furthermore, by analyzing the semantic label information of a set of similar images, we can roughly predict the category distribution of an unknown database. Therefore, deep hashing retrieval methods based on classification labels have broad application prospects. Summary of the Invention

[0005] To address the aforementioned issues in the prior art, the present invention provides a deep hash retrieval method based on classification labels. The core concept of this method is to fuse the similarity information between image pairs with the semantic information of each image. A classification loss and a metric learning loss function are used to train the network parameters, minimizing both the similarity loss and classification label loss of the generated features. A hash code structure that simultaneously preserves both semantic label information and image visual content information is then used to rapidly retrieve a test image set in Hamming space.

[0006] The present invention is achieved through the following technical solutions:

[0007] A deep hash retrieval method based on classification labels includes the following steps:

[0008] Step 1: Use a pre-trained convolutional neural network to extract high-dimensional features of images in the training image database;

[0009] Step 2: Input the high-dimensional features of the image extracted in step 1 into the deep hash network to calculate the low-dimensional class hash code;

[0010] Step 3: Input the low-dimensional class hash code obtained in step 2 into the classifier to obtain classification features;

[0011] Step 4: Calculate the loss function value using the low-dimensional features obtained in step 2 and the semantic features obtained in step 3;

[0012] Step 5: Adjust the parameters of the convolutional neural network and deep hashing network through back propagation and shared weights to obtain the final parameters of the network;

[0013] Step 6: In the test phase, the query image and the test image set are input into the final network, the obtained label is encoded and concatenated with the calculated hash code, and the Hamming distance is calculated for sorting to obtain a list of images related to the query image. Compared with the existing technology, the present invention has the following advantages:

[0014] 1. We propose a novel deep hashing network architecture that enables rapid retrieval and accurate classification of large-scale remote sensing images within a unified framework. Unlike existing deep hashing network-based methods, this approach fuses similarity information between image pairs with the semantic information of each image to produce more discriminative feature representations, resulting in better retrieval and classification results.

[0015] 2. The network parameters are trained by combining classification loss and metric learning loss functions, so that the similarity loss and classification label loss of the generated features are smaller, while the hash codes generated for preserving the visual information content and the hash codes for preserving the semantic label information are more representative.

[0016] 3. A hash code structure that can simultaneously preserve semantic label information and image visual content information is used to quickly retrieve the test image set in the Hamming space. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The network structure and process of the training phase of the present invention

[0018] Figure 2 This is the network structure and process of the test phase of the present invention

[0019] Figure 3 This is a schematic diagram of the embedding space of the classification method of the present invention.

[0020] Figure 4 This is a schematic diagram of the embedding space of the metric learning method of the present invention.

[0021] Figure 5 This is the generation process of the hash code based on the category label of the present invention DETAILED DESCRIPTION

[0022] The technical solution of the present invention will be further described below with reference to the accompanying drawings. Figure 1 and Figure 2 The overall structure and flow of the network are shown.

[0023] The deep hash retrieval method based on classification labels proposed in the present invention includes the following steps:

[0024] Step 1: Use a pre-trained convolutional neural network to extract high-dimensional features of images in the training image database.

[0025] The present invention uses the Inception network pre-trained on the ImageNet dataset to extract high-dimensional features of images. For the N input image sets {x1,…,x N}, through nonlinear transformation r i =f(x i ;ω) calculate the high-dimensional depth features {r1,…,r N}, where ω represents the parameter value of the Inception network, which is gradually optimized to the value that best suits the current data set as the network is trained.

[0026] Step 2: Input the high-dimensional features of the image extracted in step 1 into the deep hash network to calculate the low-dimensional class hash code.

[0027] Use a deep hash network constructed with a fully connected layer to perform feature dimensionality reduction to obtain low-dimensional hash code features in Represents the parameters of the deep hashing network, which are gradually optimized as the entire network is trained.

[0028] Step 3: Input the low-dimensional class hash code obtained in step 2 into the classifier to obtain classification features.

[0029] The classifier consists of a fully connected layer with a softmax function, which is used to calculate the category probability of the image. Its calculation formula is p i =softmax(u i ; ε), where ε is the parameter of the fully connected classification layer, which is optimized along with the optimization of the network parameters.

[0030] Step 4: Calculate the loss function value using the low-dimensional features obtained in step 2 and the semantic features obtained in step 3.

[0031] The metric loss that utilizes the similarity information between images and the classification loss that utilizes the semantic label information of images are combined. The difference between the two methods in the feature space is as follows: Figure 3 and Figure 4 .

[0032] The classification cross entropy loss is calculated after the classification layer to reduce the loss between the predicted class label and the actual label. The calculation formula is as follows:

[0033]

[0034] where p i Is the current picture x i The probability of belonging to each category after the classifier, p i ∈T C , <·> represents the inner product operation, and N is the number of input images. By optimizing the loss L1, the network can learn a more accurate classification probability vector for each image, thereby obtaining more representative semantic label information.

[0035] However, the classification loss can only monitor the classification accuracy of a single image. It cannot control the similarity distribution between image pairs, that is, it cannot distinguish different categories of hash codes well. Therefore, the metric loss is introduced.

[0036] In order to disperse samples of different categories while bringing samples of the same category closer, the similarity metric loss function calculates the loss of all samples in the training batch. The calculation formula is:

[0037]

[0038] P + Represents the positive proxy set corresponding to the current data, represents a set of samples similar to the agent, Represents a sample set that is different from the agent, α p and α n They are used to adjust the optimization direction of positive and negative samples respectively, so that positive and negative samples are optimized in the optimal direction. p is the threshold between positive sample pairs, which stipulates that the similarity between positive samples should be higher than this threshold, δ n is the threshold between negative sample pairs, which stipulates that the similarity between negative samples should be lower than this threshold, δ p and δ n Controls the degree of dispersion between samples. Represents sample feature u i With positive agent characteristics u p The cosine similarity between Represents sample feature u i With negative proxy feature u n The cosine similarity between .

[0039] The loss function is mainly used to learn representative features, and the class hash code will lose some information in the process of quantization into the hash code. In addition, the existence of discrete values ​​makes it difficult to calculate the derivative. Therefore, the class hash feature is used to calculate the similarity before quantization. The calculation formula is:

[0040]

[0041] Where K represents the length of the hash code. and Represents the floating-point features before hash code quantization And proxy p2. At the same time, in order to reduce the loss between the class hash code and the binary hash code, the quantization loss is introduced, and its calculation formula is:

[0042]

[0043] is the hash code of the i-th class, is the i-th hash code, which is given by the formula h K =sgn(d K ) to quantize the class hash code, where sgn(·) is the sign function, which returns the sign of a variable, which is 1 for positive values ​​and -1 for negative values. N is the training batch size. Represents the l2 norm vector to reduce the distance between the class hash code and the hash code.

[0044] L2=L p-loss +L b-loss

[0045] The final form of the loss function is:

[0046] L3=ηL1+(1-η)L2

[0047] Where η∈[0,1] is a parameter used to balance label information and similarity information. Specifically, when η=1, the loss function only uses the label information of each image; when η=0, the loss function only uses the similarity information of each image.

[0048] Step 5: Adjust the parameters of the convolutional neural network and deep hashing network through back propagation and shared weights to obtain the final parameters of the network.

[0049] In this step, the well-known back-propagation algorithm is used to adjust the global parameters of the deep hashing network. The AdamW optimizer is used, the initial learning rate is set to 0.0001, the training batch samples are randomly selected, and the size is set to 90. The value of the parameter η is shown in Table 1 and is finally set to 0.2.

[0050] Table 1: Comparison results of different hyperparameters η on the UCMD dataset

[0051]

[0052]

[0053] Step 6: For the test phase, the query image and the test image set are input into the final network, the obtained label is encoded and concatenated with the calculated hash code, and the Hamming distance is calculated for sorting to obtain a list of images related to the query image. The hash code is generated as follows Figure 5 As shown, in the test phase, the predicted probability distribution vector p of length C is generated. i Afterwards, use c i =argmax(p i ) to obtain the predicted label c of the current image i , for the predicted label c i The binary representation is used to obtain the label binary code, whose length is

[0054] The hash code that saves the visual content of the image itself is the hash network after passing b i =sgn(u i ), assuming K is the length of the hash code finally obtained, b i The length is

[0055] The final classification-based hash code is given by Generate, where Represents a vector concatenation operation. Part of the hash code is used to store label information, and the other part is used to store the image content information.

[0056] The hash codes generated by the test image set and the hash codes generated by the query image are calculated and sorted to obtain a list of images related to the query image.

[0057] The experimental results of this method are shown in Tables 2 and 3.

[0058] Table 2: Retrieval accuracy of the proposed method on the UCMD dataset

[0059]

[0060] Table 3: Retrieval accuracy of the present invention on the AID dataset

[0061]

[0062] On the UCMD dataset, when the hash code length is 16, 32, 48, and 64 bits, the retrieval accuracy is 98.97, 99.34, 99.54, and 99.60, respectively.

[0063] On the AID dataset, when the hash code length is 16, 32, 48, and 64 bits, the retrieval accuracy is 94.75, 98.08, 98.93, and 99.02, respectively.

[0064] The retrieval accuracy is improved to varying degrees compared with other methods.

[0065] Moreover, this method can obtain the semantic label results of the input image, and can complete the classification task while realizing the retrieval task.

[0066] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention. Those skilled in the art should understand that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A deep hash retrieval method based on classification labels, characterized by: The image retrieval method includes the following steps: Step 1: Use a pre-trained convolutional neural network to extract high-dimensional features of images in the training image database; Step 2: Input the high-dimensional features of the image extracted in step 1 into the deep hash network to calculate the low-dimensional class hash code; Step 3: Input the low-dimensional class hash code obtained in step 2 into the classifier to obtain classification features; Step 4: Calculate the loss function value using the low-dimensional features obtained in step 2 and the semantic features obtained in step 3; The loss function is defined as: p i Is the current picture x i The probability of belonging to each category after the classifier, p i ∈T C , <·> represents the inner product operation, N is the number of input images; L2=L p-loss +L b-loss in: P + Represents the positive proxy set corresponding to the current data, represents a set of samples similar to the agent, Represents a sample set that is different from the agent, α p and α n They are used to adjust the optimization direction of positive and negative samples respectively, so that positive and negative samples are optimized in the optimal direction. p is the threshold between positive sample pairs, which stipulates that the similarity between positive samples should be higher than this threshold, δ n is the threshold between negative sample pairs, which stipulates that the similarity between negative samples should be lower than this threshold, δ p and δ n Controls the degree of dispersion between samples. Represents sample feature u i With positive agent characteristics u p The cosine similarity between Represents sample feature u i With negative proxy feature u n cosine similarity between ; is the hash code of the i-th class, is the i-th hash code, which is given by the formula h K =sgn(d K ) is used to quantize the class hash code, where sgn(·) is a sign function that returns the sign of a variable, which is 1 for positive values ​​and -1 for negative values; N is the training batch size; represents the l2 norm vector to reduce the distance between the class hash code and the hash code; The final loss function is: L3=ηL1+(1-η)L2; Step 5: Adjust the parameters of the convolutional neural network and deep hashing network through back propagation and shared weights to obtain the final parameters of the network; Step 6: For the test phase, the query image and the test image set are input into the final network, the obtained labels are encoded and concatenated with the calculated hash codes, and the Hamming distance is calculated for sorting to obtain a list of images related to the query image.

2. A deep hash retrieval method based on classification labels according to claim 1, characterized in that After extracting high-dimensional features through the pre-trained network, the method further includes: After pre-training the network, a deep hash network is constructed to perform feature dimensionality reduction operations to extract similarity information, obtain class hash codes, and obtain low-dimensional features; A classification network is built after the deep hashing network to extract semantic information and obtain classification features.

3. A deep hash retrieval method based on classification labels according to claim 2, characterized in that After obtaining the semantic features and low-dimensional features, the loss function is calculated using these two features at the same time.

4. A deep hash retrieval method based on classification labels according to claim 1, characterized in that In step 6, the method for obtaining the image list during the testing phase is as follows: Step 1: Use the deep hash network to calculate the high-dimensional deep features to obtain low-dimensional class hash code features Step 2: Use formula b i =sgn(u i ) Calculate and obtain a visual content hash code generated based on similarity; Step 3: Using c i =argmax(p i ) to obtain the predicted label c of the current image i , for the predicted label c i Perform binary representation to obtain the label binary code, and concatenate to obtain the classification-based hash code; Step 4: Calculate the Hamming distance between the classification-based hash codes of the query image and the test image set to obtain a sorted sequence.

Images