A multi-label image retrieval method based on unsupervised deep image set hashing

Abstract

The present invention provides a multi-label image retrieval method based on unsupervised deep image set hashing, and relates to the field of image retrieval technology. The method converts a multi-label data set obtained in image retrieval into a multi-label image set data set, and the pictures in each set have exactly the same labels; uses a VGG‑16 network to extract image features from the multi-label image set data set; calculates set features for the image features of the second fully connected layer fc7 layer; uses the set features of all sets to construct a semantic structure; uses a square loss function to learn hash coding to obtain a trained network structure model; and uses the trained network structure model to implement multi-label image retrieval. The present invention can better retain the characteristic that multiple images in the same set represent the same object, while avoiding one-to-one comparison between all image features, greatly reducing the number of comparisons, and has the characteristics of reliability and speed, and is suitable for image retrieval scenarios of large-scale data.

Description

Technical Field

[0001] The present invention relates to the field of image retrieval technology, and in particular to a multi-label image retrieval method based on unsupervised deep image set hashing. Background Art

[0002] With the rapid development of network technology, the popularity of digital devices and mobile devices (smartphones, tablets, laptops), and the popularity of various websites and apps, more and more users like to share different types of content on the Internet, such as text, images, audio, and video. This phenomenon has caused the Internet data to not only be large in quantity, but also to increase explosively, and the widely used image data is also the same, which has created a demand for accurate and fast retrieval of image data. Therefore, image retrieval technology has become a research hotspot in the field of computer vision, and it is of great significance and application value in the big data environment. For example, in the medical field, given a patient's radiological image, the system can retrieve images with similar features and patterns in a large radiological image archive, which will help doctors make diagnoses and specify treatment plans. In the field of criminal investigation, given a photo of a suspect, the system can retrieve faces with similar features and attributes in a large facial database, which will help the police screen and identify suspects. In the field of e-commerce, given a user's item picture, the system can retrieve goods with similar features and categories in a large commodity image library, which will help users select and find the items they need.

[0003] The hash algorithm can map an image into a binary code that retains semantic information. The purpose of fast image retrieval can be achieved by quickly retrieving the binary code, that is, the required image can be searched in millions of images in a constant time, so it is more suitable for scenarios with large-scale data sets. Since the currently popular deep learning method has a powerful image representation ability, compared with the existing traditional hashing method, the deep hashing method formed by combining the two further improves the accuracy of hash coding and makes retrieval more accurate. However, most deep hashing methods describe images as a single subject, and its label information is also unique. However, images in reality contain multiple subjects and need to use multiple labels to express their complex semantics. Therefore, it is necessary to propose a set of deep hashing methods to solve the problem of multi-label image retrieval, so that when performing image retrieval, it is more accurate and faster, and continuously meets the growing needs.

[0004] The Chinese patent "CN202110381057.0 Multi-label image retrieval method and device based on deep hash energy model" obtains the hash code of the query image through the model file of the deep neural network pre-trained by the energy model, and uses pruning and scanning based on the Hamming distance in the retrieval stage. The Chinese patent "CN110457514A A multi-label image retrieval method based on deep hashing" introduces the cosine distance of paired multi-label image vectors as supervisory information to participate in model training, uses residual networks to extract features from multi-label images, and introduces a binary coding mechanism to reduce the dimensionality of the extracted high-dimensional features, and uses a multi-label image dataset to train a deep hash model based on a residual network.

[0005] The Chinese patent "CN110457514A A multi-label image retrieval method based on deep hashing" encodes each image separately, ignoring the fact that multiple images in the same set can represent the same object or person. In the scenario of large-scale image retrieval, one-by-one comparison is required, which will slow down the retrieval speed and fail to achieve good performance. In addition, the technical method described in the Chinese patent "CN202110381057.0 Multi-label image retrieval method and device based on deep hash energy model" is a supervised process, but in real life, large-scale image data sets often do not have label information. If label information is to be used, a lot of manpower and time are required for manual annotation, and the accuracy of manual annotation cannot be guaranteed. Therefore, it is necessary to be able to learn hash codes without labels, that is, without supervision. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a multi-label image retrieval method based on unsupervised deep image set hashing in response to the above-mentioned deficiencies in the prior art. A new unsupervised deep set hashing model suitable for multi-label image retrieval scenarios is adopted, and image sets are used as input to reduce the number of comparisons in the retrieval, thereby improving the accuracy and speed of the retrieval while ensuring accuracy.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A multi-label image retrieval method based on unsupervised deep image set hashing, comprising the following steps:

[0009] Step 1: Convert the multi-label dataset obtained from image retrieval into a multi-label image set dataset, where the images in each set have exactly the same labels.

[0010] Step 2: Use the VGG-16 network to extract image features from the multi-label image collection dataset.

[0011] Step 3: Calculate the set features of the second fully connected layer fc7 image features. For each image set, represent the image features of all images in the set as a unique set feature, while retaining as much information from different images as possible. While retaining the underlying image feature distribution, derive statistics to describe the global geometric properties of the set feature distribution.

[0012] Step 4: Use the collective features of all sets to construct the semantic structure.

[0013] Step 5: Use the square loss function to learn the hash code and obtain the trained network structure model.

[0014] Step 6: Use the trained network structure model to perform multi-label image retrieval.

[0015] The beneficial effect of adopting the above technical solution is that the multi-label image retrieval method based on unsupervised deep image set hashing provided by the present invention learns the unique binary code of a given image set by combining set representation and hash learning in a deep neural network. On the basis of extracting image features by the CNN module, the set features can statistically aggregate these features to capture set-specific distribution information, ultimately making sets from the same class have a smaller distance and sets from different classes have a larger distance. Compared with the hashing method of a single image, the present invention can better retain the characteristics of multiple images in the same set representing the same object, while avoiding one-to-one comparison between all image features, greatly reducing the number of comparisons, and has the characteristics of reliability and speed, and is suitable for image retrieval scenarios of large-scale data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flow chart of a multi-label image retrieval method based on unsupervised deep image set hashing provided by an embodiment of the present invention;

[0017] Figure 2 An example diagram of an image set data set provided by an embodiment of the present invention;

[0018] Figure 3 A schematic diagram of the set characteristics provided by an embodiment of the present invention;

[0019] Figure 4 A statistical schematic diagram of the cosine distance provided by an embodiment of the present invention; wherein FIG. (4a) is a distance distribution histogram, FIG. (4b) is a normal distribution curve diagram of the left half, and FIG. (4c) is a normal distribution curve diagram of the right half;

[0020] Figure 5 A network structure diagram provided for an embodiment of the present invention;

[0021] Figure 6 A schematic diagram of a practical application of the method provided by an embodiment of the present invention;

[0022] Figure 7 The retrieval comparison time result diagram of different methods provided in the embodiments of the present invention on two data sets NUS-WIDE and FLICKR25K; wherein Figure (7a) is the retrieval comparison time of different methods on the data set NUS-WIDE, and Figure (7b) is the retrieval comparison time of different methods on the data set FLICKR25K. DETAILED DESCRIPTION

[0023] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0024] like Figure 1 As shown, the multi-label image retrieval method based on unsupervised deep image set hashing in this embodiment is described as follows.

[0025] Step 1: Convert the multi-label dataset obtained from image retrieval into a multi-label image set dataset, where the images in each image set have exactly the same labels.

[0026] Step 1.1: Obtain the commonly used public multi-label datasets NUS-WIDE and FLICKR-25K in the field of image retrieval. Both are collected from the Flickr website. The NUS-WIDE dataset contains 269,648 public web images, each of which is annotated with one or more class labels from 81 classes. A subset of images associated with the 21 most common labels is used, totaling 195,384 images, and each label is associated with at least 5,000 images; the FLICKR-25K dataset contains 25,000 images, each of which is annotated with one or more of the 24 labels.

[0027] Step 1.2: Combine the original single labels in different quantities, and use the labels obtained by the combination to construct an image collection dataset containing collection information. The pictures in each image collection have exactly the same multi-label information. A specific example of an image collection dataset is as follows: Figure 2 shown.

[0028] Step 2: Use the VGG-16 network to extract image features from the multi-label image collection dataset.

[0029] Step 2.1: The VGG-16 network consists of 5 convolutional layers (conv1-conv5), 3 fully connected layers (fc6-fc8), and a softmax output layer. The layers are separated by max-pooling. Many hashing methods have achieved good performance using this network. Initialize the network parameters of the VGG-16 model pre-trained on the ImageNet dataset.

[0030] Step 2.2: Use the image features extracted by the second fully connected layer fc7 in the VGG-16 network as input for subsequent steps. The second fully connected layer fc7 in the VGG-16 network can extract high-dimensional features containing rich semantic information.

[0031] Step 3: Calculate the set features of the second fully connected layer fc7 image features. In order to ensure the universality of the set features, for each image set, use a suitable method to represent the image features of all images in the set as a unique set feature, while retaining as much information from different images as possible. While retaining the underlying image feature distribution, derive statistics to describe the global geometric properties of the set feature distribution.

[0032] Specifically, given an input image set S, the average feature mean(S), feature variance var(S), minimum feature min(S), and maximum feature max(S) are calculated respectively. Figure 3 As shown in the figure, it can be seen that these statistical information can well independently represent the distribution characteristics of the set and describe the overall characteristics of the set. mean(S) represents the average position of the set in the sample feature space, var(S) represents the change of the value of each feature in the given set S relative to its mean, and min(S) and max(S) are used to represent the value range of the set feature. The set feature F(S) of set S is the direct concatenation of the above four features, that is:

[0033] F(S)=concatenation(mean(S),var(S),min(S),max(S)) (1)

[0034] Step 4: Use the collective features of all sets to construct the semantic structure.

[0035] Step 4.1: Cosine distance can be used to measure the similarity between two vectors. The closer the cosine distance is to 1, the more similar they are, and the closer the cosine distance is to 0, the less similar they are. Calculate the cosine distance between each pair of set feature data points, and then draw a distance distribution histogram based on the cosine distance, such as Figure 4 As shown in Figure (4a) in , since the cosine distances of most data point pairs are relatively large, the shape of the distance distribution histogram is similar to the normal distribution.

[0036] Step 4.2: Use the maximum value of the distribution probability to divide the distance distribution histogram into two parts, left and right, and calculate the mean μ of the left and right parts respectively. left , μ right and standard deviation σ left , σ right , and are fitted into appropriate normal distribution curves, such as Figure 4 As shown in Figures (4b) and (4c) in .

[0037] Step 4.3: Set the distance threshold d1 = μ left -ασ left and d2=μ right +βσ right , where α and β are hyperparameters that control the proportion of each type of data. Here, α = 1 and β = 0.5 are set. The distance distribution histogram is divided into three segments according to the distance threshold, and the semantic structure D is constructed. For details, see formula (2). d(i, j) represents the cosine distance between the set features of set i and set j. When D ij When D is 1, it means that the set features of the two sets are highly similar (that is, they almost share all the labels). ij When it is 0, it means that the set characteristics of the two sets are moderately similar. ij When it is -1, it means that the similarity of the set features of the two sets is low (they share a small number of labels).

[0038]

[0039] Step 5: Use the square loss function to learn the hash code.

[0040] Step 5.1: Use hash codes to construct similar structures.

[0041] The hash layer can map the set features into binary vectors, which have higher retrieval efficiency and take up less storage space than floating-point feature vectors. The hash layer is added after the set feature calculation. The output of the hash layer is expressed as: h(F(S)) = s(WF(S) + b), where s(·) is a nonlinear activation function, where W and b represent the parameters of the hash layer, and F(S) represents the result of the set feature calculation in step 3. Then the sign function is applied to obtain the hash code of the data: b i =sign(h(F(S))),b i ∈{-1,+1} K , represents the hash code of set i, and K represents the length of the hash code. For hash codes, the inner product can be used to calculate the number of the same corresponding bits of the hash code, thereby reflecting the semantic similarity. Therefore, based on the hash code, a similarity structure H is defined using the inner product:

[0042]

[0043] Step 5.2: Model training.

[0044] Since the method of this embodiment is based on image collections, for the image collection dataset constructed on NUS-WIDE in step 1, 700 image collections are randomly selected to form a training set, 400 image collections are randomly selected and 5 images are extracted from each collection to form a test query set of size 2000. For the image collection dataset built on FLICKR25K, 500 image collections are randomly selected to form a training set, 200 image collections are randomly selected and 5 images are extracted from each collection to form a test query set of size 1000. The remaining image collections in each dataset are used as retrieval databases. The method of this embodiment is implemented using open source Pytorch, and the model is optimized by SGD. When loading the data in the collection, the batch size is set to 16, and when training, the batch size is set to 4. The size of the training image is resized to 224×224 as input, and the learning rate is fixed to 0.001.

[0045] Use the square loss function to learn the hash code and get the network structure model as follows Figure 5 As shown in the figure, the model takes an image set as input and uses the convolutional neural network VGG-16 to extract image features. Then, the set features are calculated using low-level distribution information such as the mean, standard deviation, maximum value, and minimum value, and all image features in the set are converted into a unique set feature representation. A semantic structure D is constructed using the set features to capture the semantic relationship between images, and a similar structure H is constructed using hash codes calculated using geometric features. The image feature extraction and hash function learning are guided by minimizing the difference between the semantic structure D and the similar structure H. However, since the sign function acts on the network output to obtain a discrete result, the back propagation of the network is not differentiable. Therefore, the tanh function is used instead of the sign function to perform the relaxation operation, resulting in a continuously differentiable loss function, where D ij and H ij They represent the semantic structure D obtained by statistical semantic analysis of the data and the (i, j)th element of the similar structure H learned by the hash code, n is the number of sets, and the square loss function is specifically expressed as:

[0046]

[0047] in, c i =tanh(h(F(S))),c i The value of each dimension of is between -1 and 1.

[0048] Step 6: Use the trained network structure model to implement multi-label image retrieval based on unsupervised deep image set hashing.

[0049] After the model training is completed, multi-label image retrieval is performed. Figure 6 As shown, first, the hash codes of all images in the multi-label image set dataset and the hash codes of all image sets are obtained through the trained network structure model to build a hash code database; the hash code of the image to be retrieved is obtained using the trained network structure model, and the hash code of the image to be retrieved is compared with the hash code database to obtain a candidate set of retrieval results; the distance between the hash code of the image to be retrieved and the image in the candidate set is calculated, and re-sorted to obtain the final retrieval result, thereby achieving the goal of multi-label image retrieval.

[0050] The following is a comparison of several methods to evaluate the above model.

[0051] The method provided in this embodiment is compared with several typical hashing methods, including LSH, SH, ITQ, SpH and SSDH, which are all hashing methods based on single image encoding. For these hashing methods based on single image encoding, 2000 images are randomly selected as the test query set and 10000 images are randomly selected as the training set on NUS-WIDE, and 1000 images are randomly selected as the test query set and 5000 images are randomly selected as the training set on FLICKR25K. The remaining images in the two data sets are used as query databases. The mean average precision (MAP) and the retrieval time t are used. avg The specific formula of MAP is formula (5). As a more important and objective evaluation indicator in retrieval, it is defined as the average of all average precisions (AP) generated by each unique query image, where AP is the area under the PR curve in each query instance; t avg The calculation formula is shown in formula (6), which is defined as the average retrieval comparison time of multiple images, where m represents the number of retrieval tests, and t i0 Indicates the hash code compared to the previous time, t i1 Indicates the time after the comparison ends.

[0052]

[0053]

[0054] As shown in Table 1, the performance of these methods on the NUS-WIDE and FLICKR25K datasets is shown, and the MAP results of these methods on the datasets when the hash code length ranges from 16 bits to 128 bits are shown; it can be seen that the model proposed in this embodiment performs better than all the comparison methods at different hash code lengths. Compared with SSDH on these two datasets, the model proposed in this embodiment obtains an improvement of 0.03%-1.79% in MAP of different bits, indicating that the use of image collections can improve the accuracy of unsupervised hashing methods in multi-label retrieval. Figure 7 is the retrieval comparison time of these methods when the hash code length is 32 bits. Compared with other benchmark methods, the hash code comparison time required by the model proposed in this embodiment during the retrieval process is much lower than that of other methods, indicating that using image sets as the input of the model can effectively reduce the number of comparisons, thereby shortening the hash code comparison time. Comparing the results of the model proposed in this embodiment on the two data sets, the average comparison time on the FLICKR25K data set is about 0.27ms, which is better than 1ms on the NUS-WIDE data set. This shows that the effect of the model proposed in this embodiment in shortening the hash code comparison time is related to the size of the data set and the number of sets.

[0055] Table 1 Comparison of MAP results on NUS-WIDE and FLICKR25K with different hash code lengths

[0056]

[0057] The multi-label image retrieval method based on unsupervised deep image set hashing provided in this embodiment uses the image set as the input of the deep hash model. By extracting image features and calculating set features, the set is converted into a unique representation, and then a semantic structure is constructed based on the statistical information of the set features, and the difference between it and the similar structure constructed by the hash code is minimized, thereby completing the hash code learning process. Different from previous methods, this embodiment applies the image set-based method to unsupervised deep hash multi-label image retrieval for the first time, which greatly shortens the comparison time of hash codes and is suitable for large-scale image retrieval. Experiments show that in multi-label image retrieval, the deep hash algorithm based on image sets is better than the deep hash algorithm based on single images.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A multi-label image retrieval method based on unsupervised deep image set hashing, characterized by: The method comprises the following steps: Step 1: Convert the multi-label dataset obtained from image retrieval into a multi-label image set dataset, where the images in each set have exactly the same labels; Step 2: Use the VGG-16 network to extract image features from the multi-label image set dataset; the VGG-16 network includes 5 convolutional layers, 3 fully connected layers and a softmax output layer, and the layers are separated by max-pooling; the network parameters of the VGG-16 model pre-trained on the ImageNet dataset are used for initialization; Use the image features extracted by the second fully connected layer fc7 in the VGG-16 network as input for subsequent steps; Step 3: Calculate the set features of the image features of the second fully connected layer fc7; for each image set, represent the image features of all images in the set as a unique set feature, while retaining as much information from different images as possible; while retaining the distribution of the underlying image features, derive statistics to describe the global geometric properties of the set feature distribution; Given an input image set , respectively calculate the average feature , characteristic variance , minimum feature , the biggest feature , Representing a collection The average position in the sample feature space, Represents a given set The change of each feature value relative to its mean, , Used to indicate the value range of set characteristics; set The collection characteristics of The average feature , characteristic variance , minimum feature , the biggest feature Direct concatenation of four features, namely: ; Step 4: Use the collective features of all sets to construct the semantic structure; Step 5: Use the square loss function to learn the hash code and obtain the trained network structure model; Step 6: Use the trained network structure model for multi-label image retrieval.

2. The multi-label image retrieval method based on unsupervised deep image set hashing according to claim 1, characterized in that: In step 1, after obtaining the multi-label dataset in image retrieval, the screened original single labels are combined in different quantities, and the combined labels are used to construct an image dataset containing set information, that is, a multi-label image set dataset, and the pictures in each image set have exactly the same multi-label information.

3. The multi-label image retrieval method based on unsupervised deep image set hashing according to claim 1 is characterized in that: The specific method of step 4 is: Step 4.1: Calculate the cosine distance between each pair of set feature data points, and draw a distance distribution histogram based on the cosine distance; Step 4.2: Use the maximum value of the distribution probability to divide the distance distribution histogram into two parts, left and right, and calculate the mean of the left and right parts respectively. , and standard deviation , , and are fitted into appropriate normal distribution curves respectively; Step 4.3: Set distance threshold and , , , according to the distance threshold, the distance distribution histogram is divided into three segments to construct the semantic structure , as shown in formula (2); （2） in, Representing a collection and the cosine distance between the set features of set j; when When it is 1, it means that the set characteristics of the two sets are highly similar; when When it is 0, it means that the set characteristics of the two sets are moderately similar; when When it is -1, it means that the similarity of the set features of the two sets is low.

4. The multi-label image retrieval method based on unsupervised deep image set hashing according to claim 3 is characterized in that: The specific method of step 5 is: Step 5.1: Use hash codes to construct similar structures; The hash layer is added after the set feature calculation, and the output of the hash layer is expressed as: ,in, is a nonlinear activation function, and Represents the parameters of the hash layer, F ( S ) represents the result of the set feature calculation in step 3; application Symbol function to get the hash code of the data: , , which represents the set i The hash code of K Indicates the length of the hash code; Define a similar structure based on the hash code using the inner product H : （3） Step 5.2: Perform model training; For the image set dataset constructed in step 1, randomly select several image sets to form a training set, then randomly select several image sets and extract several images from each set to form a test query set, and the remaining image sets are used as a retrieval database; The square loss function is used to learn the hash code. The square loss function is specifically expressed as: (4) in, and They represent the semantic structures obtained by statistical semantic analysis of the data. Similar structure to that learned by hash codes No. elements, is the number of sets, , , The value of each dimension of is between -1 and 1. Function to perform relaxation operation; After learning the above hash code, the trained network structure model is obtained.

5. The multi-label image retrieval method based on unsupervised deep image set hashing according to claim 4 is characterized in that: The specific method of step 6 is: Obtain hash codes of all images in the multi-label image set dataset and hash codes of all image sets through the trained network structure model, and build a hash code database; The trained network structure model is used to obtain the hash code of the image to be retrieved, and the hash code of the image to be retrieved is compared with the hash code database to obtain a candidate set of retrieval results; The distance between the hash codes of the image to be retrieved and the candidate set images is calculated and re-sorted to obtain the final retrieval result.

6. The multi-label image retrieval method based on unsupervised deep image set hashing according to claim 4, characterized in that: In step 5.2, when loading the data in the image collection, the batch size is set to 16, and during training, the batch size is set to 4; the size of the training image is adjusted to 224×224 as input, and the learning rate is fixed to 0.001.

Images