Unsupervised depth image hash retrieval method based on external knowledge

Abstract

The invention provides an unsupervised depth image Hash retrieval method based on external knowledge, and belongs to the technical field of image processing.The method comprises the steps that an image data set containing multiple feature categories is obtained, and a noun data set containing daily English noun vocabularies is collected; image features and noun features are extracted respectively, corresponding unique text features are generated for each image by using the noun features, and the unique text features are external knowledge; respectively constructing and training an image feature hash network and a text feature hash network; based on a trained image feature hash network, binary hash codes of query and to-be-retrieved image data are extracted, and the retrieval accuracy of the image data is evaluated by calculating and sorting Hamming distances between the hash codes. According to the method, the problem that in the prior art, similarity information extracted only depending on depth visual features has an inherent upper limit is solved, and the problem that inaccurate information is easily introduced by a comparative learning strategy is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an unsupervised deep image hash retrieval method based on external knowledge. Background Art

[0002] With the explosive growth of multimedia data on the Internet, high retrieval efficiency and low storage cost have become one of the common requirements for large-scale information retrieval tasks. Hash-based methods can convert high-dimensional continuous features into compact discrete binary hash codes. This method can greatly speed up the calculation speed and save storage space through XOR operations while maintaining the semantic features of the original data. In addition, in large-scale information retrieval tasks, costly manual annotation is difficult to deploy in actual application scenarios, which limits the application scope of supervised hashing technology. Therefore, in the absence of label information, how to guide the hash model to learn effective similarity information from the semantic features of the data itself has become the focus of research, namely the deep unsupervised hashing method. In recent years, due to its characteristics of no need for costly data annotation, high computational efficiency and economical storage, deep unsupervised hashing has become one of the widely studied directions in image retrieval tasks.

[0003] Existing deep unsupervised hashing methods can be roughly divided into three categories: similarity reconstruction-based methods, pseudo-label-based methods, and prediction-free self-supervised learning-based methods.

[0004] In similarity reconstruction-based methods, a pre-trained deep learning model (such as VGG-F) is usually used to extract deep visual features from the original image data. These features are then used to reconstruct the semantic similarity structure between images using specific methods such as manifold similarity, local similarity, and cosine similarity. The purpose is to reflect the true similarity between images in the feature space. Then, guided by this reconstructed similarity structure, the hash model is optimized so that the hash code it generates can effectively maintain the similarity between the data, that is, similar images should have similar hash codes, while different images should have significantly different hash codes.

[0005] The second category of methods is based on pseudo-labeling techniques, which usually use a deep clustering framework to generate pseudo-labels. This process involves clustering unlabeled data and treating the clustering results as pseudo-labels. These pseudo-labels are regarded as a kind of semantic information to guide the subsequent hash code generation process. In this way, the model can be guided to learn meaningful and distinguishable hash representations, thereby improving the accuracy of retrieval.

[0006] Methods based on unpredicted self-supervised learning mainly rely on contrastive learning, generative adversarial networks and other methods to improve the quality of hash codes. The third type of method usually integrates contrastive learning mechanisms into unsupervised hashing methods, and imposes constraints on hash codes by modeling the relationship between positive samples (similar samples) and negative samples (dissimilar samples). This strategy helps to generate more discriminative hash codes, and can achieve efficient data retrieval and matching even in the absence of labeled information.

[0007] Existing deep unsupervised hashing methods mainly rely on the deep visual features of images and combine different learning strategies such as pseudo-labeling and contrastive learning to guide the generation of hash codes. However, relying solely on deep visual features to obtain similarity information has certain limitations: (1) Although strategies such as data augmentation and contrastive learning are used, the information capacity of deep visual features (visual information) is limited, which will lead to inaccurate extracted similarity information. Especially when dealing with similar images with large visual differences, the generated hash codes may reduce the accuracy of retrieval. (2) Deep unsupervised hashing methods based on pseudo-labeling and contrastive learning strategies are prone to introduce inaccurate information. When dealing with complex datasets, if the initial clustering results are poor, a large number of incorrect labels are likely to appear in the generated pseudo-labels. At the same time, contrastive learning usually regards different views of the same image as positive samples and all other images as negative samples. This strategy will mistakenly judge some samples that actually belong to the same category as negative samples, thereby interfering with the optimization process of hash codes. Summary of the invention

[0008] In view of the above-mentioned deficiencies in the prior art, the present invention provides an unsupervised deep image hash retrieval method based on external knowledge, which solves the problem that the similarity information extracted by relying solely on deep visual features in the prior art has an inherent upper limit, and solves the problem that contrastive learning strategies easily introduce inaccurate information.

[0009] In order to achieve the above objectives, the technical solution adopted by the present invention is: an unsupervised deep image hash retrieval method based on external knowledge, comprising the following steps:

[0010] S1. Obtain an image dataset containing multiple object categories and collect a noun dataset covering daily English noun vocabulary;

[0011] S2, based on the unlabeled image dataset and the noun dataset, extract image features and noun features respectively, and use the noun features to generate corresponding unique text features for each image, wherein the unique text features are external knowledge;

[0012] S3. Based on the paired image features and text features, an image feature hash network and a text feature hash network are constructed and trained respectively, and under the guidance of the text feature hash code generated by the text feature hash network, the image feature hash network is optimized to obtain a trained image feature hash network;

[0013] S4. Based on the trained image feature hash network, the binary hash codes of the query and image data to be retrieved are extracted, and the retrieval accuracy of the image data is evaluated by calculating and sorting the Hamming distance between the hash codes to complete the hash retrieval of unsupervised deep images.

[0014] The beneficial effects of the present invention are as follows: the present invention utilizes abundant and easily available noun data, and through a special external knowledge construction method, generates more reliable and discriminative text features (external knowledge), and forms paired image-text features with deep image features to train hash networks, thereby capturing more similarity information. In addition, the present invention can construct supervisory information for positive and negative sample pairs with the help of similarity information between external knowledge, guide comparative learning strategies, and generate more discriminative hash codes.

[0015] Furthermore, the S1 comprises the following steps:

[0016] S101, obtaining a number of natural image datasets with accurate labels, and obtaining a noun dataset N covering daily English noun vocabulary, wherein the natural image dataset includes various objects in life scenes;

[0017] S102, splitting the natural image dataset into a training set T, a query set Q, and a retrieval set R, wherein the natural images in the training set T have their label information removed, and the natural images in the query set Q and the retrieval set R have their label information retained;

[0018] S103: For the training set T, two different data augmentation views V are generated by random scaling, cropping, flipping and color transformation. 1 and V 2 ;

[0019] S104, respectively, the original images of the training set T, the query set Q and the retrieval set R are processed by a unified scaling and cropping operation to obtain a preprocessed training data set X t , query dataset X q And retrieve dataset X r , and discard the training data set X t The label information in the training dataset is used to obtain an unlabeled training dataset, and the image dataset containing multiple object categories is obtained. The training dataset X t , query dataset X q and retrieve dataset X rImage data in, and two data augmentation views V 1 and V 2 The dimensions are the same.

[0020] The beneficial effects of the above further solution are as follows: the present invention improves the generalization ability of the model by data augmentation, and uses the query data set X q and retrieve dataset X r Different augmentation methods can prevent the model from overfitting during training.

[0021] Furthermore, the S2 comprises the following steps:

[0022] S201, construct and use the pre-training parameters to initialize the visual language pre-training model to obtain the image feature extraction model M L And text feature extraction model M T ;

[0023] S202, for the preprocessed training data set X t The image data in and two different data augmentation views V for each image data 1 and V 2 , using the image feature extraction model M L Extract features and obtain image features I of the training set i And the corresponding data-enhanced image features and

[0024] S203: for the noun data set N, generate multiple corresponding texts for each noun data, and extract the generated texts through the text feature extraction model M. T And calculate the average value to get the text feature T corresponding to each noun data j ;

[0025] S204, calculate each text feature T j The L2 distance between them is used to delete redundant text features with an L2 distance of 0;

[0026] S205, taking the text features with the largest distance L2 obtained by screening as set X t1 , and cluster the text features except for deleting redundancy using clustering algorithm;

[0027] S206: The text features that are farthest from the cluster center are selected as the set X t2 , the text features closest to the cluster center are selected as the set X t3 ;

[0028] S207, based on set X t1 , set X t2and the set X t3 , get the text feature set X after filtering out the redundant t ;

[0029] S208, set the text feature set X t The text feature T′ in j Image features I of the training set i Perform weighted summation to obtain each image feature I i The corresponding unique text feature E i , where the unique text feature E i Enhance image features for data and The corresponding unique text feature.

[0030] The beneficial effect of the above further scheme is: the present invention makes the weighted summed text features more reliable and discriminative by screening out redundant text features, and can thus be more effectively used as supervisory information to guide the training of the image feature hashing network.

[0031] Furthermore, the unique text feature E i The expression is as follows:

[0032]

[0033] Among them, w i,j Represents each text feature T′ j Relative to each image feature I i The weight of, cos(.,.) represents the cosine similarity calculation, τ 1 Represents the temperature parameter, N T' represents the total number of text features after removing redundancy, T′ n Represents the text feature set X t All text features in .

[0034] The beneficial effect of the above further solution is that: by weighted summation, the unique text feature corresponding to each image feature is calculated, so that this text feature can be focused on integrating the text feature set X t All the features in the image can be better matched with the image features.

[0035] Furthermore, the S3 comprises the following steps:

[0036] S301, construct and initialize the image feature hash network H respectively v and text feature hashing network H e , where the image feature hashing network H v and text feature hashing network H eBoth include two fully connected layers and one hyperbolic tangent tanh nonlinear layer;

[0037] S302, using the following formula, in the unique text feature E i The cosine similarity is calculated between them, and the images corresponding to the text features whose cosine similarity is greater than the preset threshold are taken as positive samples, and the images corresponding to the text features whose cosine similarity is less than or equal to the preset threshold are taken as negative samples. The obtained positive and negative samples will be used to guide the information noise contrast estimation loss function L info Build:

[0038] sim i,j =cos(E i ,E j )

[0039] Among them, E i and E j Respectively represent the unique text features corresponding to different image features, sim i,j Represents two text features E i and E j The cosine similarity between them, cos(.,.) represents the cosine similarity calculation;

[0040] S303, enhancing image features with data and Through the image feature hashing network H v , and get two data-enhanced image hash codes and And the unique text feature E i Through the initialized text feature hashing network H e , get the text hash code h E ;

[0041] S304, respectively calculating two data enhanced image hash codes and Between, data augmentation image hash codes and the text hash code h E Between and data augmented image hash codes and the text hash code The information noise contrast estimation loss function L info , to train image feature hashing network and text feature hashing network simultaneously;

[0042] S305, using information entropy theory, by converting the hash code in the interval [-1, 1] into a probability value in the interval [0, 1], and calculating the balance loss function L based on the probability value bal , further trained image feature hashing network.

[0043] S306. Repeat steps S303 to S305 until the image feature hash network reaches convergence, thereby completing the training of the image feature hash network.

[0044] The beneficial effect of the above further scheme is: the present invention uses a comparative learning strategy to associate the two data enhancement image features and text features, and then integrates "external knowledge" on the basis of "internal knowledge" to guide the learning of the hash network, and uses the cosine similarity between the text features corresponding to different images as the supervision information of the positive and negative sample pairs to guide the comparative learning strategy, prevent similar samples from being regarded as negative sample pairs, and generate more discriminative hash codes. At the same time, the balanced loss function L is introduced bal , so that the generated hash codes are as evenly distributed as possible.

[0045] Furthermore, the loss function L info The expression is as follows:

[0046]

[0047] Among them, L info represents the loss function of information noise contrast estimation InfoNCE, Represents two data augmented image hash codes and The information noise contrast estimation loss function between Represents a data-augmented image hash code and the text hash code h E The information noise contrast estimation loss function between Represents another data augmented image hash code and the text hash code h E The information noise contrast estimation loss function between them, N represents the total number of image samples in the training set, i represents the counting index from 1 to N, and k represents the image hash code and the text hash code It belongs to both positive and negative samples, j represents the image hash code and the text hash code belongs to the positive sample, τ 2 represents the temperature parameter, and Both represent image features I i The corresponding data-augmented image hash code, and Both represent text features E i The corresponding text hash code.

[0048] The technical effect of the above further scheme is: by using the information noise contrast to estimate the loss function, the contrast between positive samples (similar samples) and negative samples (dissimilar samples) is maximized, thereby helping image hash codes and text hash codes to learn more discriminative feature representations.

[0049] Furthermore, the balance loss function L bal The expression is as follows:

[0050]

[0051] Where bit represents the total number of bits of the hash code, [b] represents the bth bit in the total number of bits of the hash code, and Represents hash code Hash Code and hash code The bth position of Represents all image hash codes The hash code obtained after the bitwise average operation is Represents a data-augmented image feature The extracted image hash code, Represents all image hash codes The hash code obtained after the bitwise average operation is Represents another data augmented image feature The extracted image hash code, Represents the hash code of all texts in the training set The hash code obtained after the bitwise average operation is Represents the image feature I i The corresponding unique text feature E i The extracted text hash code, N represents the total number of image samples in the training set, and i represents the counting index from 1 to N.

[0052] The technical effect brought about by the above further scheme is: by balancing the loss function and utilizing the information entropy theory, the distribution of each bit of the hash code after bit-wise averaging processing is made as even as possible, so that when the image feature hash network and the text feature hash network face an unbalanced data set, the hash codes generated by the network will not be too biased towards a larger category as a whole.

[0053] Furthermore, the S4 comprises the following steps:

[0054] S401, using image feature extraction model M L , respectively extract the query data set X q and retrieve dataset X r Image features in Iq and image feature I r ;

[0055] S402, image feature I q and image feature I r , respectively input into the trained image feature hashing network H v In the one-to-one correspondence, we get the query image hash code h q and retrieve the image hash code h r ;

[0056] S403, using the Hamming distance to calculate the query image hash code h q and retrieve the image hash code h r The Hamming distance is calculated and sorted from small to large based on the Hamming distance. The top K sorted results are used as the retrieval results. The MAP@K value calculated using the sorted results is used as the retrieval accuracy of the image data to complete the hash retrieval of unsupervised deep images. MAP@K represents the average of the average accuracy of the top K retrieval results of all queries, which is used to measure the ranking quality of related retrieval results.

[0057] The technical effect of the above further solution is: by calculating the average MAP@K of the average accuracy of the first K retrieval results of all queries, the query data set X is simulated. q Image pair retrieval dataset X r The process of retrieving images in the image repository is then used to effectively evaluate the retrieval quality of the unsupervised deep image hashing retrieval method.

[0058] Furthermore, the expression of the Hamming distance is as follows:

[0059]

[0060] Among them, d H (h q ,h r ) represents the query image hash code h q and retrieve the image hash code h r The Hamming distance between them, [b] represents the bth bit in the total number of bits of the hash code, bit represents the total number of bits of the hash code, h q [b] and h r [b] represent the query image hash code h q and retrieve the image hash code h r The bth position of Represents the indicator function, indicating the query image hash code h q and retrieve the image hash code h r If the bth bit of is not equal, it returns 1; otherwise, it returns 0.

[0061] The technical effect of the above further solution is: by calculating the query image hash code h q and retrieve the image hash code h r The Hamming distance between them can provide a ranking basis for calculating the average value MAP@K of the average precision (AveragePrecision) of the first K retrieval results of all queries.

[0062] Furthermore, the expression of the MAP@K value is as follows:

[0063]

[0064]

[0065] Where |Q| represents the total number of queries, q represents one query among all queries, AP(q) represents the average accuracy of the top K retrieval results of a query q, and r q represents the total number of related items, represents the indicator function, which means that it returns 1 when the item at the kth position is relevant, otherwise it returns 0. P(k) represents the ratio of the number of relevant contents in the first K search results to the total number. Truepositives@K represents the number of relevant contents in the first K search results, and falsepositives@K represents the number of irrelevant contents in the first K search results.

[0066] The technical effect brought about by the above further solution is: by calculating the average value MAP@K of the average precision (Average Precision) of the first K retrieval results of all queries, the accuracy and relevance of the retrieval results can be reflected, and then the retrieval quality of the image retrieval method can be evaluated. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0068] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0069] Example

[0070] In view of the shortcomings of existing deep unsupervised hashing methods, the present invention proposes an unsupervised deep image hash retrieval method based on external knowledge. In order to solve the problem that there is an inherent upper limit to the similarity information extracted only by deep visual features, the present invention defines the visual information directly obtained from the image data as "internal knowledge". At the same time, compared with the detailed mining of similarity information from limited "internal knowledge", the present invention chooses to introduce "external knowledge" in textual modality, because such knowledge is usually rich and easy to obtain, and can better assist the hash model in learning the semantic similarity between images. Specifically, the present invention mines the potential "external knowledge" of deep visual features, combines "internal knowledge" with "external knowledge", so as to more accurately capture the similarity information between images, thereby providing more comprehensive knowledge when generating hash codes, and making up for the limitations brought about by relying solely on "internal knowledge".

[0071] In addition, in order to solve the problem that contrastive learning strategies easily introduce inaccurate information, the present invention uses "external knowledge" as additional supervisory information. By analyzing the similarity between "external knowledge" features, the present invention can more accurately define the positive and negative sample relationships between different images during contrastive learning. For example, images with a higher degree of similarity to "external knowledge" in the text modality will be regarded as positive samples, and images with a lower degree of similarity will be regarded as negative samples. With the guidance of "external knowledge", a more discriminative hash code can be generated.

[0072] like Figure 1 As shown, the present invention provides an unsupervised deep image hash retrieval method based on external knowledge, and its implementation method is as follows:

[0073] S1. Obtain an image dataset containing multiple object categories and collect a noun dataset covering daily English noun vocabulary. The implementation method is as follows:

[0074] S101, obtaining a number of natural image datasets with accurate labels, and obtaining a noun dataset N covering daily English noun vocabulary, wherein the natural image dataset includes various objects in life scenes;

[0075] In this embodiment, a large number of natural image datasets with accurate labels are obtained, which contain various objects in life scenes as much as possible. At the same time, the noun vocabulary part in the 'WordNet' vocabulary is obtained as the noun dataset N.

[0076] S102, splitting the natural image dataset into a training set T, a query set Q, and a retrieval set R, wherein the natural images in the training set T have their label information removed, and the natural images in the query set Q and the retrieval set R have their label information retained;

[0077] S103: For the training set T, two different data augmentation views V are generated by random scaling, cropping, flipping and color transformation. 1 and V 2 ;

[0078] S104, respectively, the original images of the training set T, the query set Q and the retrieval set R are processed by a unified scaling and cropping operation to obtain a preprocessed training data set X t , query dataset X q And retrieve dataset X r , and discard the training data set X t The label information in the training dataset is used to obtain an unlabeled training dataset, and the image dataset containing multiple object categories is obtained. The training dataset X t , query dataset X q and retrieve dataset X r Image data in, and two data augmentation views V 1 and V 2 The dimensions are the same.

[0079] S2. Based on the unlabeled image dataset and the noun dataset, image features and noun features are extracted respectively, and the noun features are used to generate corresponding unique text features for each image, where the unique text features are external knowledge, and the implementation method is as follows:

[0080] S201, construct and use the pre-training parameters to initialize the visual language pre-training model to obtain the image feature extraction model M L And text feature extraction model M T ;

[0081] S202, for the preprocessed training data set X t The image data in and two different data augmentation views V for each image data 1 and V 2 , using the image feature extraction model M L Extract features and obtain image features I of the training set i And the corresponding data-enhanced image features and

[0082] S203: for the noun data set N, generate multiple corresponding texts for each noun data, and extract the generated texts through the text feature extraction model M. T And calculate the average value to get the text feature T corresponding to each noun data j ;

[0083] In this embodiment, for the noun dataset N, the text is composed of "a photo of the large [Noun]", "a photo of the small [Noun]" and "a bad photo of the [Noun]", and the text is extracted through the text feature extraction model M. T After averaging, we get the text feature T corresponding to each noun data. j .

[0084] S204, calculate each text feature T j The L2 distance between them is used to delete redundant text features with an L2 distance of 0;

[0085] S205, taking the text features with the largest distance L2 obtained by screening as set X t1 , and cluster the text features except for deleting redundancy using clustering algorithm;

[0086] S206: The text features that are farthest from the cluster center are selected as the set X t2 , the text features closest to the cluster center are selected as the set X t3 ;

[0087] S207, based on set X t1 , set X t2 and the set X t3 , get the text feature set X after filtering out the redundant t ;

[0088] In this embodiment, each text feature T is first calculated j The L2 distance between them is used to delete the redundant text features with an L2 distance of 0, and the text features with the largest L2 distance are selected as the set X. t1 In addition, the Kmeans clustering algorithm is used to cluster the remaining text features into 300 cluster centers, and the outlier text features far from the cluster centers are selected as the set X t2 At the same time, select the 5 text features closest to each cluster center as set X t3 , and then get the text feature set X after removing the redundant t ={X t1 ,X t2 ,X t3}.

[0089] S208, set the text feature set X t The text feature T′ in j Image features I of the training set i Perform weighted summation to obtain each image feature I iThe corresponding unique text feature E i , where the unique text feature E i Enhance image features for data and The corresponding unique text feature.

[0090] Unique text feature E i The expression is as follows:

[0091]

[0092] Among them, w i,j Represents each text feature T′ j Relative to each image feature I i The weight of, cos(.,.) represents the cosine similarity calculation, τ 1 Represents the temperature parameter, N T′ represents the total number of text features after removing redundancy, T′ n Represents the text feature set X t All text features in .

[0093] S3. Based on paired image features and text features, an image feature hash network and a text feature hash network are constructed and trained respectively, and under the guidance of the text feature hash code generated by the text feature hash network, the image feature hash network is optimized to obtain a trained image feature hash network. The implementation method is as follows:

[0094] S301, construct and initialize the image feature hash network H respectively v and text feature hashing network H e , where the image feature hashing network H v and text feature hashing network H e Both include two fully connected layers and one hyperbolic tangent tanh nonlinear layer;

[0095] In this embodiment, the image feature hash network H v and text feature hashing network H e Both include two fully connected layers, and the dimensions of the two fully connected layers are [input dim -input dim ] and [input dim -bit dim ], where bit dim Equal to [16or32or64], a hyperbolic tangent tanh nonlinear layer is connected at the end of the fully connected layer to approximate the binary hash code.

[0096] S302, using the following formula, in the unique text feature E iThe cosine similarity is calculated between the two images, and the images corresponding to the text features whose cosine similarity is greater than the preset threshold are taken as positive samples, and the images corresponding to the text features whose cosine similarity is less than or equal to the preset threshold are taken as negative samples. The obtained positive and negative samples are used to guide the construction of the information noise contrast estimation loss function:

[0097] sim i,j =cos(E i ,E j )

[0098] Among them, E i and E j Respectively represent the unique text features corresponding to different image features, sim i,j Represents two text features E i and E j The cosine similarity between them, cos(.,.) represents the cosine similarity calculation;

[0099] In this embodiment, by calculating different text features E i and E j The cosine similarity between them is used to consider the image features corresponding to the text features with a cosine similarity greater than 0.9 as positive samples, and vice versa as negative samples, thereby guiding the construction of the information noise contrast estimation loss function in the subsequent S304.

[0100] S303, enhancing image features with data and Through the image feature hashing network H v , and get two data-enhanced image hash codes and And the unique text feature E i Through the initialized text feature hashing network H e , get the text hash code h E ;

[0101] S304, respectively calculating two data enhanced image hash codes and Between, data augmentation image hash codes and the text hash code h E Between and data augmented image hash codes and the text hash code The information noise contrast estimation loss function L info , to train image feature hashing network and text feature hashing network simultaneously;

[0102] The loss function L info The expression is as follows:

[0103]

[0104] Among them, L info represents the loss function of information noise contrast estimation InfoNCE, Represents two data augmented image hash codes and The information noise contrast estimation loss function between Represents a data-augmented image hash code and the text hash code h E The information noise contrast estimation loss function between Represents another data augmented image hash code and the text hash code h E The information noise contrast estimation loss function between them, N represents the total number of image samples in the training set, i represents the counting index from 1 to N, and k represents the image hash code and the text hash code It belongs to both positive and negative samples, j represents the image hash code and the text hash code belongs to the positive sample (denoted as j∈+), τ 2 represents the temperature parameter, and Both represent image features I i The corresponding data-augmented image hash code, and Both represent text features E i The corresponding text hash code.

[0105] S305, using information entropy theory, by converting the hash code in the interval [-1, 1] into a probability value in the interval [0, 1], and calculating the balance loss function L based on the probability value bal , further trained image feature hashing network.

[0106] In this embodiment, information entropy is a quantitative indicator to measure the uncertainty of random events. When the entropy value is larger, the corresponding distribution tends to be more uniform. In order to ensure that the sum of each hash code in a large number of random samples is close to uniform distribution, the present invention uses information entropy theory to convert the hash code in the interval [-1,1] into a probability value in the interval [0,1], and calculates the equilibrium loss function L based on the probability value. bal , so that the generated hash code can present the characteristics of uniform distribution as much as possible.

[0107] The balance loss function L bal The expression is as follows:

[0108]

[0109] Where bit represents the total number of bits of the hash code, [b] represents the bth bit in the total number of bits of the hash code, and Represents hash code Hash Code and hash code The bth position of Represents all image hash codes The hash code obtained after the bitwise average operation is Represents a data-augmented image feature The extracted image hash code, Represents all image hash codes The hash code obtained after the bitwise average operation is Represents another data augmented image feature The extracted image hash code, Represents the hash code of all texts in the training set The hash code obtained after the bitwise average operation is Represents the image feature I i The corresponding unique text feature E i The extracted text hash code, N represents the total number of image samples in the training set, and i represents the counting index from 1 to N.

[0110] S306. Repeat steps S303 to S305 until the image feature hash network reaches convergence, thereby completing the training of the image feature hash network.

[0111] In this embodiment, the image feature hash network can fully learn the characteristics of the data by repeating the training steps multiple times. Each time the training steps are repeated, the image feature hash network will update the weights through back propagation, gradually reducing the output of the loss function, thereby improving the accuracy and generalization ability of the model.

[0112] S4. Based on the trained image feature hash network, the binary hash codes of the query and image data to be retrieved are extracted, and the retrieval accuracy of the image data is evaluated by calculating and sorting the Hamming distance between the hash codes to complete the hash retrieval of unsupervised deep images. The implementation method is as follows:

[0113] S401, using image feature extraction model M L , respectively extract the query data set X q and retrieve dataset X r Image features in I q and image feature I r ;

[0114] S402, image feature I q and image feature Ir , respectively input into the trained image feature hashing network H v In the one-to-one correspondence, we get the query image hash code h q and retrieve the image hash code h r ;

[0115] S403, using the Hamming distance to calculate the query image hash code h q and retrieve the image hash code h r The Hamming distance is calculated and sorted from small to large based on the Hamming distance. The top K sorted results are used as the retrieval results. The MAP@K value calculated using the sorted results is used as the retrieval accuracy of the image data to complete the hash retrieval of unsupervised deep images. MAP@K represents the average of the average accuracy of the top K retrieval results of all queries, which is used to measure the ranking quality of related retrieval results.

[0116] Hamming distance d H (h q ,h r ) is as follows:

[0117]

[0118] Among them, d H (h q ,h r ) represents the query image hash code h q and retrieve the image hash code h r The Hamming distance between them, [b] represents the bth bit in the total number of bits of the hash code, bit represents the total number of bits of the hash code, h q [b] and h r [b] represent the query image hash code h q and retrieve the image hash code h r The bth position of Represents the indicator function, indicating the query image hash code h q and retrieve the image hash code h r If the bth bit of is not equal, it returns 1; otherwise, it returns 0.

[0119] The expression of MAP@K value is as follows:

[0120]

[0121] Where |Q| represents the total number of queries, q represents one query among all queries, AP(q) represents the average accuracy of the top K retrieval results of a query q, and r q represents the total number of related items, represents the indicator function, which means that it returns 1 when the item at the kth position is relevant, otherwise it returns 0. P(k) represents the ratio of the number of relevant contents in the first K search results to the total number. Truepositives@K represents the number of relevant contents in the first K search results, and falsepositives@K represents the number of irrelevant contents in the first K search results.

[0122] In summary, the present invention utilizes rich and easily available noun data, and through a special external knowledge construction method, generates more reliable and discriminative text features (external knowledge), and forms paired image-text features with deep image features to train hash networks, thereby capturing more similarity information. In addition, the present invention can construct supervision information for positive and negative sample pairs with the help of similarity information between external knowledge, guide comparative learning strategies, and generate more discriminative hash codes.

Claims

1. An unsupervised deep image hash retrieval method based on external knowledge, characterized in that: The following steps are involved: S1. Obtain an image dataset containing multiple object categories and collect a noun dataset covering daily English noun vocabulary; S2, based on the unlabeled image dataset and the noun dataset, extract image features and noun features respectively, and use the noun features to generate corresponding unique text features for each image, wherein the unique text features are external knowledge; S3. Based on the paired image features and text features, an image feature hash network and a text feature hash network are constructed and trained respectively, and under the guidance of the text feature hash code generated by the text feature hash network, the image feature hash network is optimized to obtain a trained image feature hash network; S4. Based on the trained image feature hash network, the binary hash codes of the query and image data to be retrieved are extracted, and the retrieval accuracy of the image data is evaluated by calculating and sorting the Hamming distance between the hash codes to complete the hash retrieval of unsupervised deep images.

2. The unsupervised deep image hash retrieval method based on external knowledge according to claim 1 is characterized in that: The S1 comprises the following steps: S101, obtaining a number of natural image datasets with accurate labels, and obtaining a noun dataset N covering daily English noun vocabulary, wherein the natural image dataset includes various objects in life scenes; S102, splitting the natural image dataset into a training set T, a query set Q, and a retrieval set R, wherein the natural images in the training set T have their label information removed, and the natural images in the query set Q and the retrieval set R have their label information retained; S103, for the training set T, generate two different data augmented views V1 and V2 by random scaling, cropping, flipping, and color transformation; S104, respectively, the original images of the training set T, the query set Q and the retrieval set R are processed by a unified scaling and cropping operation to obtain a preprocessed training data set X t , query dataset X q And retrieve dataset X r , and discard the training data set X t The label information in the training dataset is used to obtain an unlabeled training dataset, and the image dataset containing multiple object categories is obtained. The training dataset X t , query dataset X q and retrieve dataset X r The image data in has the same dimensions as the two data augmentation views V1 and V2.

3. The unsupervised deep image hash retrieval method based on external knowledge according to claim 2 is characterized in that: The S2 comprises the following steps: S201, construct and use the pre-training parameters to initialize the visual language pre-training model to obtain the image feature extraction model M L And text feature extraction model M T ; S202, for the preprocessed training data set X t The image data in the image and two different data augmentation views V1 and V2 for each image data are extracted using the image feature extraction model M L Extract features and obtain image features I of the training set i And the corresponding data-enhanced image features and S203: for the noun data set N, generate multiple corresponding texts for each noun data, and extract the generated texts through the text feature extraction model M. T And calculate the average value to get the text feature T corresponding to each noun data j ; S204, calculate each text feature T j The L2 distance between them is used to delete redundant text features with an L2 distance of 0; S205, taking the text features with the largest distance L2 obtained by screening as set X t1 , and cluster the text features except for deleting redundancy using clustering algorithm; S206: The text features that are farthest from the cluster center are selected as the set X t2 , the text features closest to the cluster center are selected as the set X t3 ; S207, based on set X t1 , set X t2 and the set X t3 , get the text feature set X after filtering out the redundant t ; S208, set the text feature set X t The text features T' j Image features I of the training set i Perform weighted summation to obtain each image feature I i The corresponding unique text feature E i , where the unique text feature E i Enhance image features for data and The corresponding unique text feature.

4. The unsupervised deep image hash retrieval method based on external knowledge according to claim 3 is characterized in that: The unique text feature E i The expression is as follows: Among them, w i,j Represents each text feature T' j Relative to each image feature I i The weight of, cos(.,.) represents the cosine similarity calculation, τ1 represents the temperature parameter, N T' represents the total number of text features after redundant features are removed, T' n Represents the text feature set X t All text features in .

5. The unsupervised deep image hash retrieval method based on external knowledge according to claim 3 is characterized in that: The S3 comprises the following steps: S301, construct and initialize the image feature hash network H respectively v and text feature hashing network H e , where the image feature hashing network H v and text feature hashing network H e Both include two fully connected layers and one hyperbolic tangent tanh nonlinear layer; S302, using the following formula, in the unique text feature E i The cosine similarity is calculated between them, and the images corresponding to the text features whose cosine similarity is greater than the preset threshold are taken as positive samples, and the images corresponding to the text features whose cosine similarity is less than or equal to the preset threshold are taken as negative samples. The obtained positive and negative samples will be used to guide the information noise contrast estimation loss function L info Build: Yes i,j =cos(E i ,AND j ) Among them, E i and E j Respectively represent the unique text features corresponding to different image features, sim i,j Represents two text features E i and E j The cosine similarity between them, cos(.,.) represents the cosine similarity calculation; S303, enhancing image features with data and Through the image feature hashing network H v , and get two data-enhanced image hash codes and And the unique text feature E i Through the initialized text feature hashing network H e , get the text hash code h E ; S304, respectively calculating two data enhanced image hash codes and Between, data augmentation image hash codes and the text hash code h E Between and data augmented image hash codes and the text hash code The information noise contrast estimation loss function L info , to train image feature hashing network and text feature hashing network simultaneously; S305, using information entropy theory, by converting the hash code in the interval [-1,1] into a probability value in the interval [0,1], and calculating the balance loss function L based on the probability value bal , further trained image feature hashing network. S306. Repeat steps S303 to S305 until the image feature hash network reaches convergence, thereby completing the training of the image feature hash network.

6. The unsupervised deep image hash retrieval method based on external knowledge according to claim 5 is characterized in that: The loss function L info The expression is as follows: Among them, L info represents the loss function of information noise contrast estimation InfoNCE, Represents two data augmented image hash codes and The information noise contrast estimation loss function between Represents a data-augmented image hash code and the text hash code h E The information noise contrast estimation loss function between Represents another data augmented image hash code and the text hash code h E The information noise contrast estimation loss function between them, N represents the total number of image samples in the training set, i represents the counting index from 1 to N, and k represents the image hash code and the text hash code It belongs to both positive and negative samples, j represents the image hash code and the text hash code belongs to the positive sample, τ2 represents the temperature parameter, and Both represent image features I i The corresponding data-augmented image hash code, and Both represent text features E i The corresponding text hash code.

7. The unsupervised deep image hash retrieval method based on external knowledge according to claim 6 is characterized in that: The balance loss function L bal The expression is as follows: Where bit represents the total number of bits of the hash code, [b] represents the bth bit in the total number of bits of the hash code, and Respectively represent hash codes Hash Code and hash code The bth position of Represents all image hash codes The hash code obtained after the bitwise average operation is Represents a data-augmented image feature The extracted image hash code, Represents all image hash codes The hash code obtained after the bitwise average operation is Represents another data augmented image feature The extracted image hash code, Represents the hash code of all texts in the training set The hash code obtained after the bitwise average operation is Represents the image feature I i The corresponding unique text feature E i The extracted text hash code, N represents the total number of image samples in the training set, and i represents the counting index from 1 to N.

8. The unsupervised deep image hash retrieval method based on external knowledge according to claim 5, characterized in that: The S4 comprises the following steps: S401, using image feature extraction model M L , respectively extract the query data set X q and retrieve dataset X r Image features in I q and image feature I r ; S402, image feature I q and image feature I r , respectively input into the trained image feature hashing network H v In the one-to-one correspondence, we get the query image hash code h q and retrieve the image hash code h r ; S403, using the Hamming distance to calculate the query image hash code h q and retrieve the image hash code h r The Hamming distance is calculated and sorted from small to large based on the Hamming distance. The top K sorted results are used as the retrieval results. The MAP@K value calculated using the sorted results is used as the retrieval accuracy of the image data to complete the hash retrieval of unsupervised deep images. MAP@K represents the average of the average accuracy of the top K retrieval results of all queries, which is used to measure the ranking quality of related retrieval results.

9. The unsupervised deep image hash retrieval method based on external knowledge according to claim 8, characterized in that: The Hamming distance d H (h q ,h r ) is as follows: Among them, d H (h q ,h r ) represents the query image hash code h q and retrieve the image hash code h r The Hamming distance between them, [b] represents the bth bit in the total number of bits of the hash code, bit represents the total number of bits of the hash code, h q [b] and h r [b] represent the query image hash code h q and retrieve the image hash code h r The bth position of Represents the indicator function, indicating the query image hash code h q and retrieve the image hash code h r If the bth bit of is not equal, it returns 1; otherwise, it returns 0.

10. The unsupervised deep image hash retrieval method based on external knowledge according to claim 9, characterized in that: The expression of the MAP@K value is as follows: Where Q represents the total number of queries, q represents one query among all queries, AP(q) represents the average accuracy of the first K retrieval results of a query q, and r q represents the total number of related items, represents the indicator function, which means that it returns 1 when the item at the kth position is relevant, otherwise it returns 0. P(k) represents the ratio of the number of relevant contents in the first K search results to the total number. Truepositives@K represents the number of relevant contents in the first K search results, and falsepositives@K represents the number of irrelevant contents in the first K search results.

Images