An Unsupervised Deep Image Hashing Retrieval Method Based on External Knowledge

Abstract

The present invention provides an unsupervised deep image hashing retrieval method based on external knowledge, belonging to the technical field of image processing. The method includes obtaining an image data set containing multiple feature categories and collecting a noun data set covering daily English noun vocabulary; respectively extracting image features and noun features, and using the noun features to generate a corresponding unique text feature for each image, where the unique text feature is external knowledge; respectively constructing and training an image feature hashing network and a text feature hashing network; based on the trained image feature hashing network, extracting the binary hash codes of the query and the image data to be retrieved, and evaluating the retrieval accuracy of the image data by calculating and sorting the Hamming distances between the hash codes. The present invention solves the problem that the similarity information relying only on the extraction of deep visual features in the prior art has an inherent upper limit, and solves the problem that the contrast learning strategy is prone to introducing inaccurate information.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to an unsupervised deep image hashing retrieval method based on external knowledge. Background Art

[0002] With the explosive growth of Internet multimedia data, high retrieval efficiency and low storage cost have become one of the common requirements for large-scale information retrieval tasks. The hashing-based method can convert high-dimensional continuous features into compact discrete binary hash codes. On the basis of maintaining the semantic features of the original data, the calculation speed can be greatly accelerated through XOR operation, saving storage space. In addition, in large-scale information retrieval tasks, costly manual annotation is difficult to be deployed in actual application scenarios, which limits the application scope of supervised hashing technology. Therefore, in the absence of label information, how to guide the hashing model to learn effective similarity information from the semantic features of the data itself has become the research focus, that is, the deep unsupervised hashing method. In recent years, due to its characteristics of no need to consume cost for data annotation, high calculation efficiency and economical storage, deep unsupervised hashing has become one of the widely studied directions in image retrieval tasks.

[0003] The existing deep unsupervised hashing methods can be roughly divided into three categories: similarity reconstruction-based methods, pseudo-label-based methods, and prediction-free self-supervised learning-based methods.

[0004] In the similarity reconstruction-based methods, a pre-trained deep learning model (such as VGG-F) is usually used to extract deep visual features from the original image data. Subsequently, specific methods such as manifold similarity, local similarity, and cosine similarity are used to utilize these features to reconstruct the semantic similarity structure between images, aiming to reflect the true similarity between images in the feature space. Then, guided by this reconstructed similarity structure, the hashing model is optimized so that the generated hash codes can effectively maintain the similarity between data, that is, similar images should have similar hash codes, while different images should have significantly different hash codes.

[0005] The second type of method is the pseudo-label-based technology, which usually uses a deep clustering framework to generate pseudo-labels. This process involves clustering the unlabeled data and regarding the clustering result as a pseudo-label. These pseudo-labels are regarded as a kind of semantic information to guide the subsequent hash code generation process. In this way, the model can be guided to learn meaningful and distinguishable hash representations, thereby improving the retrieval accuracy.

[0006] Methods based on unsupervised self-supervised learning mainly rely on methods such as contrastive learning and generative adversarial networks to improve the quality of hash codes. The third type of method usually integrates a contrastive learning mechanism into unsupervised hashing methods. By modeling the relationship between positive samples (similar samples) and negative samples (dissimilar samples), constraints are imposed on the hash codes. This strategy helps to generate more discriminative hash codes, enabling efficient data retrieval and matching even in the absence of labeled information.

[0007] Existing deep unsupervised hashing methods mainly rely on the deep visual features of images and combine different learning strategies such as pseudo-labels and contrastive learning to guide the generation of hash codes. However, relying solely on deep visual features to obtain similarity information has certain limitations: (1) Although strategies such as data augmentation and contrastive learning are adopted, due to the limited information capacity of deep visual features (visual information) themselves, the extracted similarity information may not be accurate enough. Especially when dealing with homogeneous images with large visual differences, the generated hash codes may reduce the retrieval accuracy. (2) Deep unsupervised hashing methods based on pseudo-label and contrastive learning strategies are prone to introducing inaccurate information. When dealing with complex datasets, if the initial clustering results are poor, there are likely to be a large number of incorrect labels in the generated pseudo-labels. At the same time, contrastive learning usually regards different views of the same image as positive samples and all other images as negative samples. This strategy may misclassify some samples that actually belong to the same category as negative samples, thus interfering with the optimization process of hash codes. Summary of the Invention

[0008] In view of the above deficiencies in the prior art, the present invention provides an unsupervised deep image hashing retrieval method based on external knowledge, which solves the problem that the similarity information extracted only relying on deep visual features in the prior art has an inherent upper limit, and solves the problem that the contrastive learning strategy is prone to introducing inaccurate information.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is: an unsupervised deep image hashing retrieval method based on external knowledge, including the following steps:

[0010] S1. Obtain an image dataset containing multiple object categories and collect a noun dataset covering common English noun vocabulary;

[0011] S2. Based on the unlabeled image dataset and the noun dataset, extract image features and noun features respectively, and use the noun features to generate a corresponding unique text feature for each image, where the unique text feature is external knowledge;

[0012] S3. Based on the paired image features and text features, construct and train an image feature hashing network and a text feature hashing network respectively, and optimize the image feature hashing network under the guidance of the text feature hash code generated by the text feature hashing network to obtain the trained image feature hashing network;

[0013] S4. Based on the trained image feature hashing network, extract the binary hash codes of the query and the image data to be retrieved, and evaluate the retrieval accuracy of the image data by calculating and sorting the Hamming distances between the hash codes, thus completing the hashing retrieval of unsupervised depth images.

[0014] The beneficial effects of the present invention are as follows: The present invention utilizes rich and easily obtainable noun data, and through a special construction method of external knowledge, generates more reliable and discriminative text features (external knowledge), which form paired image-text features with depth image features to train the hashing network, thereby capturing more similarity information. In addition, the present invention can construct the supervision information of positive and negative sample pairs with the help of the similarity information between external knowledge, guide the contrast learning strategy, and generate more discriminative hash codes.

[0015] Further, the S1 includes the following steps:

[0016] S101. Obtain a number of natural image data sets with accurate labels, and obtain a noun data set N covering daily English noun vocabulary. Among them, the natural image data set includes various objects in life scenes;

[0017] S102. Split the natural image data set into a training set T, a query set Q, and a retrieval set R. Among them, the labels of the natural images in the training set T are removed, and the labels of the natural images in the query set Q and the retrieval set R are retained;

[0018] S103. For the training set T, generate two different data augmentation views V1 and V2 through random scaling, cropping, flipping, and color transformation;

[0019] S104. Respectively process the original images of the training set T, the query set Q, and the retrieval set R through unified scaling and cropping operations to obtain the preprocessed training data set X t 、query data set X q and retrieval data set X r , and at the same time, discard the label information in the training data set X t to obtain an unlabeled training data set, and complete the acquisition of the image data set containing multiple object categories. Among them, the image data in the training data set X t 、query data set X q and retrieval data set X r have the same dimensions as the two data augmentation views V1 and V2.

[0020] The beneficial effects of the above further solution are as follows: The present invention improves the generalization ability of the model through data augmentation, and uses different augmentation methods for the query dataset X q and the retrieval dataset X r to prevent the model from overfitting during training.

[0021] Furthermore, the S2 includes the following steps:

[0022] S201. Construct and initialize the vision-language pre-trained model with pre-trained parameters to obtain the image feature extraction model M L and the text feature extraction model M T ;

[0023] S202. For the image data in the pre-processed training dataset X t and two different data augmentation views V1 and V2 of each image data, use the image feature extraction model M L to extract features, and obtain the image features I i of the training set and the corresponding data-augmented image features and

[0024] S203. For the noun dataset N, generate multiple corresponding texts for each noun data, pass the generated texts through the text feature extraction model M T and calculate the average value to obtain the text feature T j corresponding to each noun data;

[0025] S204. Calculate the L2 distance between each text feature T j and delete the redundant text features with an L2 distance of 0;

[0026] S205. Use the text feature with the largest L2 distance obtained by screening as the set X t1 , and use the clustering algorithm to cluster the text features except for deleting the redundancy;

[0027] S206. Use the text feature with the farthest distance from the clustering center obtained by screening as the set X t2 , and use the text feature with the closest distance from the clustering center obtained by screening as the set X<http: / / www.example.com / t3 ;

[0028] S207. Based on the set X t1 , the set X t2 and the set X t3 , obtain the text feature set X t after screening out the redundancy;

[0029] S208. Weightedly sum the text feature T' in the text feature set X t with the image feature I of the training set j to obtain the unique text feature E corresponding to each image feature I i , where the unique text feature E i is the unique text feature corresponding to the data-augmented image feature i and i . and <000**********

[0030] The beneficial effect of the above further solution is that the present invention makes the text feature after weighted summation more reliable and discriminative by screening out redundant text features, and thus can be more effectively used as supervision information to guide the training of the image feature hashing network.

[0031] Furthermore, the expression of the unique text feature E i is as follows:

[0032]

[0033] where w i,j represents the weight of each text feature T' j relative to each image feature I i , cos(.,.) represents the cosine similarity calculation, τ1 represents the temperature parameter, N T' represents the total number of text features after screening out redundancy, and T' n represents all text features in the text feature set X t .

[0034] The beneficial effect of the above further solution is that by weighted summation, the unique text feature corresponding to each image feature is calculated, so that this text feature can focus on fusing all features in the text feature set X t to better match the image feature.

[0035] Furthermore, S3 includes the following steps:

[0036] S301. Respectively construct and initialize the image feature hashing network H v and the text feature hashing network H e , where both the image feature hashing network H v and the text feature hashing network H e include two fully connected layers and one hyperbolic tangent tanh non-linear layer;

[0037] S302. Using the following formula, on the unique text feature E i ​Perform cosine similarity calculation among them, and use the images corresponding to the text features with cosine similarity greater than the preset threshold as positive samples, and use the images corresponding to the text features less than or equal to the preset threshold as negative samples. Among them, the obtained positive samples and negative samples will be used to guide the construction of the information noise contrast estimation loss function L info :

[0038] sim i,j = cos(E i , E j )

[0039] Among them, E i and E j respectively represent the unique text features corresponding to different image features. sim i,j represents the cosine similarity between two text features E i and E j , and cos(.,.) represents cosine similarity calculation;

[0040] S303. Pass the data-augmented image features and through the image feature hashing network H v to obtain two data-augmented image hash codes and and pass the unique text feature E i through the initialized text feature hashing network H e to obtain the text hash code h E ;

[0041] S304. Calculate the information noise contrast estimation loss function L and between the two data-augmented image hash codes, between the data-augmented image hash code and the text hash code h E and between the data-augmented image hash code and the text hash code to train the image feature hashing network and the text feature hashing network simultaneously; info

[0042] S305. Utilize the information entropy theory to convert the hash codes in the interval [-1, 1] into probability values in the interval [0, 1], and calculate the balanced loss function L bal to further train the image feature hashing network.

[0043] S306. Repeat steps S303 to S305 until the image feature hashing network converges, thus completing the training of the image feature hashing network.

[0044] ​The beneficial effects of the above further solution are as follows: Through the contrastive learning strategy, the present invention correlates the features of two data-augmented pictures and text features, and then incorporates "external knowledge" on the basis of "internal knowledge" to guide the learning of the hash network. The cosine similarity between the text features corresponding to different images is used as the supervision information for positive and negative sample pairs to guide the contrastive learning strategy, preventing similar samples from being regarded as negative sample pairs, and thus generating more discriminative hash codes. At the same time, a balanced loss function L bal is introduced to make the generated hash codes as evenly distributed as possible.

[0045] Furthermore, the expression of the loss function L info is as follows:

[0046]

[0047] where L info represents the loss function of InfoNCE (Information Noise Contrastive Estimation), represents the information noise contrastive estimation loss function between the hash codes of two data-augmented images and ; represents the information noise contrastive estimation loss function between the hash code of one data-augmented image and the text hash code h E ; represents the information noise contrastive estimation loss function between the hash code of another data-augmented image and the text hash code h E ; N represents the total number of image samples in the training set, i represents the counting index from 1 to N, k represents that the image hash code and the text hash code belong to both positive samples and negative samples, j represents that the image hash code and the text hash code belong to positive samples, τ2 represents the temperature parameter, and both represent the hash codes of the data-augmented images corresponding to the picture feature I i , and both represent the text hash codes corresponding to the text feature E i .

[0048] The technical effects brought by the above further solution are as follows: By using the information noise contrastive estimation loss function, the contrast between positive samples (similar samples) and negative samples (dissimilar samples) is maximized, thereby helping the image hash code and the text hash code to learn more discriminative feature representations.

[0049] Furthermore, the balanced loss function L bal has the following expression:

[0050]

[0051] where bit represents the total number of bits of the hash code, and [b] represents the b-th bit among the total number of bits of the hash code. and respectively represent the b-th bit of the hash code hash code and the hash code ; represents the hash code obtained after the bitwise average operation on all image hash codes ; represents an image hash code extracted from a data-augmented image feature ; represents the hash code obtained after the bitwise average operation on all image hash codes ; represents an image hash code extracted from another data-augmented image feature ; represents the hash code obtained after the bitwise average operation on all text hash codes in the training set ; represents the unique text feature E i corresponding to the picture feature I i extracted text hash code, N represents the total number of image samples in the training set, and i represents the counting index from 1 to N.

[0052] The technical effect brought by the above further solution is that through the balanced loss function and using the information entropy theory, the distribution of the hash code after bitwise average processing is as uniform as possible in each bit, so that when the image feature hash network and the text feature hash network face a dataset with class imbalance, the generated hash code will not be overly biased towards the class with a larger number as a whole.

[0053] Furthermore, S4 includes the following steps:

[0054] S401: Use the image feature extraction model M L to extract the image feature I q in the query dataset X r and the image feature I q in the retrieval dataset X r one by one respectively;

[0055] S402: Input the image feature I q and the image feature I r into the trained image feature hash network Hv In this process, the query image hash code h is obtained in a one-to-one correspondence q and the retrieved image hash code h r ;

[0056] S403. Calculate the Hamming distance between the query image hash code h q and the retrieved image hash code h r , sort them in ascending order based on the magnitude of the Hamming distance, and take the top K sorting results as the retrieval results. At the same time, take the MAP@K value calculated using the sorting results as the retrieval accuracy of the image data, thus completing the hash retrieval of unsupervised depth images. Here, MAP@K represents the average of the average accuracies of the top K retrieval results for all queries and is used to measure the sorting quality of relevant retrieval results.

[0057] The technical effect brought by the above further solution is: By calculating the average MAP@K of the average accuracies of the top K retrieval results for all queries, the process of retrieving the images in the retrieval dataset X q using the images in the query dataset X r is simulated, and thus the retrieval quality of the unsupervised depth image hash retrieval method is effectively evaluated.

[0058] Furthermore, the expression of the Hamming distance is as follows:

[0059]

[0060] where d H (h q , h r ) represents the Hamming distance between the query image hash code h q and the retrieved image hash code h r . [b] represents the b-th bit in the total number of bits of the hash code, bit represents the total number of bits of the hash code, h q [b] and h r [b] respectively represent the b-th bit of the query image hash code h q and the retrieved image hash code h r . represents the indicator function, indicating that it returns 1 when the b-th bits of the query image hash code h q and the retrieved image hash code h r are not equal, otherwise, it returns 0.

[0061] The technical effect brought by the above further solution is: By calculating the query image hash code h q and the retrieved image hash code h rThe Hamming distance between them can provide a sorting basis for calculating the mean Average Precision (MAP@K) of the top K retrieval results for all queries.

[0062] Furthermore, the expression of the MAP@K value is as follows:

[0063]

[0064]

[0065] Where |Q| represents the total number of queries, q represents one query among all queries, AP(q) represents the average precision of the top K retrieval results of a query q, and r q represents the total number of relevant items, represents the indicator function, which returns 1 when the item at the k-th position is relevant, otherwise, it returns 0. P(k) represents the ratio of the number of relevant contents to the total number in the top K retrieval results, truepositives@K represents the number of relevant contents in the top K retrieval results, and falsepositives@K represents the number of irrelevant contents in the top K retrieval results.

[0066] The technical effect brought by the above further solution is that by calculating the mean MAP@K of the average precision (Average Precision) of the top K retrieval results for all queries, the accuracy and relevance of the retrieval results can be reflected, and further, the retrieval quality of the image retrieval method can be evaluated. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a flowchart of the method of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0068] The following describes the specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions created using the concept of the present invention are within the scope of protection.

[0069] Embodiment

[0070] Aiming at the deficiencies of existing deep unsupervised hashing methods, the present invention proposes an unsupervised deep image hashing retrieval method based on external knowledge. To solve the problem that the similarity information extracted solely from deep visual features has an inherent upper limit, the present invention defines the visual information directly obtained from image data as "internal knowledge". At the same time, compared with exhaustively mining similarity information from limited "internal knowledge", the present invention chooses to introduce "external knowledge" in the text modality because such knowledge is usually rich and easy to obtain, and can better assist the hashing model in learning the semantic similarity between images. Specifically, the present invention combines "internal knowledge" with "external knowledge" by mining the potential "external knowledge" of deep visual features to more accurately capture the similarity information between images, thereby providing more comprehensive knowledge when generating hash codes and making up for the limitations brought by solely relying on "internal knowledge".

[0071] In addition, to solve the problem that the contrast learning strategy is prone to introducing inaccurate information, the present invention uses "external knowledge" as additional supervision information. By analyzing the similarity between "external knowledge" features, the present invention can more accurately define the positive and negative sample relationships between different images during the contrast learning process. For example, images with high similarity in the "external knowledge" of the text modality are regarded as positive samples, and images with low similarity are regarded as negative samples, and with the guidance of "external knowledge", more discriminative hash codes are generated.

[0072] As Figure 1 shown, the present invention provides an unsupervised deep image hashing retrieval method based on external knowledge, and its implementation method is as follows:

[0073] S1. Obtain an image data set containing multiple object categories, and collect a noun data set covering the vocabulary of daily English nouns. The implementation method is as follows:

[0074] S101. Obtain several natural image data sets with accurate labels, and obtain a noun data set N covering the vocabulary of daily English nouns. Among them, the natural image data set includes various objects in life scenes;

[0075] In this embodiment, a large number of natural image data sets with accurate labels are obtained, covering as many various objects in life scenes as possible. At the same time, the noun vocabulary part in the 'WordNet' thesaurus is obtained as the noun data set N.

[0076] S102. Split the natural image data set into a training set T, a query set Q, and a retrieval set R. Among them, the labels of the natural images in the training set T are removed, and the labels of the natural images in the query set Q and the retrieval set R are retained;

[0077] S103. For the training set T, generate two different data augmentation views V1 and V2 through random scaling and cropping, flipping, and color transformation;

[0078] S104. Respectively process the original images of the training set T, the query set Q, and the retrieval set R through unified scaling and cropping operations to obtain the preprocessed training data set X t , query data set X q and retrieval data set X r , and at the same time, discard the label information in the training data set X t to obtain an unlabeled training data set, and complete the acquisition of an image data set containing multiple object categories. Among them, the image data in the training data set X t , query data set X q and retrieval data set X r have the same dimensions as the two data augmentation views V1 and V2.

[0079] S2. Based on the unlabeled image data set and the noun data set, extract image features and noun features respectively, and use the noun features to generate corresponding unique text features for each image. Among them, the unique text features are external knowledge, and the implementation method is as follows:

[0080] S201. Construct and initialize a vision-language pre-trained model with pre-trained parameters to obtain an image feature extraction model M L and a text feature extraction model M T ;

[0081] S202. For the image data in the preprocessed training data set X t and two different data augmentation views V1 and V2 of each image data, use the image feature extraction model M L to extract features to obtain the image features I i of the training set and the corresponding data augmentation image features and

[0082] S203. For the noun data set N, generate multiple corresponding texts for each noun data, and pass the generated texts through the text feature extraction model M T and calculate the average value to obtain the text feature T j ;

[0083] In this embodiment, for the noun data set N, texts are formed in the way of "a photo of the large [Noun]", "a photo of the small [Noun]", and "a bad photo of the [Noun]", and the texts are passed through the text feature extraction model MT After averaging, the text feature T corresponding to each noun data is obtained j .

[0084] S204. Calculate the L2 distance between each text feature T j , and delete the redundant text features with an L2 distance of 0;

[0085] S205. Take the text feature with the largest L2 distance obtained by screening as the set X t1 , and use the clustering algorithm to cluster the text features except for the deleted redundancy;

[0086] S206. Take the text feature with the farthest distance from the clustering center obtained by screening as the set X t2 , and take the text feature with the closest distance from the clustering center obtained by screening as the set X t3 ;

[0087] S207. Based on the set X t1 , the set X t2 , and the set X t3 , the text feature set X after screening redundancy is obtained t ;

[0088] In this embodiment, first calculate the L2 distance between each text feature T j , delete the redundant text features with an L2 distance of 0, and screen out the text features with a relatively large nearest L2 distance as the set X t1 . In addition, use the Kmeans clustering algorithm to cluster the remaining text features into 300 clustering centers, screen out the outlier text features with a relatively far distance from the clustering center as the set X t2 , and at the same time, select the 5 text features with the closest distance to each clustering center as the set X t3 , and then obtain the text feature set X after removing redundancy t ={X t1 ,X t2 ,X t3}.

[0089] S208. Perform weighted summation processing on the text feature T' t in the text feature set X j and the image feature I i of the training set to obtain the unique text feature E i corresponding to each image feature I i , where the unique text feature E i is the unique text feature corresponding to the data augmentation image feature and .

[0090] Unique text feature E i The expression is as follows:

[0091]

[0092] where w i,j represents the weight of each text feature T' j relative to each image feature I i cos(.,.) represents the cosine similarity calculation, τ1 represents the temperature parameter, N T′ represents the total number of text features after removing redundancy, and T' n represents all text features in the text feature set X t in.

[0093] S3. Based on the paired image features and text features, construct and train an image feature hashing network and a text feature hashing network respectively, and optimize the image feature hashing network under the guidance of the text feature hash code generated by the text feature hashing network to obtain the trained image feature hashing network. The implementation method is as follows:

[0094] S301. Construct and initialize an image feature hashing network H v and a text feature hashing network H e , where the image feature hashing network H v and the text feature hashing network H e both include two fully connected layers and one hyperbolic tangent tanh non-linear layer;

[0095] In this embodiment, the image feature hashing network H v and the text feature hashing network H e both include two fully connected layers, and the dimensions of the two fully connected layers are [input dim - input dim and [input dim - bit dim , where bit dim is equal to [16 or 32 or 64]. A hyperbolic tangent tanh non-linear layer is connected at the end of the fully connected layer to approximate the binary hash code.

[0096] S302. Use the following formula to calculate the cosine similarity between the unique text features E i . Take the images corresponding to the text features with cosine similarity greater than the preset threshold as positive samples, and take the images corresponding to the text features less than or equal to the preset threshold as negative samples. The obtained positive samples and negative samples are used to guide the construction of the information noise contrast estimation loss function:

[0097] sim i,j = cos(Ei , E j )

[0098] Among them, E i and E j respectively represent the unique text features corresponding to different image features, and sim i,j represents the cosine similarity between two text features E i and E j . cos(.,.) represents the calculation of cosine similarity;

[0099] In this embodiment, by calculating the cosine similarity between different text features E i and E j , the image features corresponding to the text features with a cosine similarity greater than 0.9 are regarded as positive samples, and vice versa as negative samples, so as to guide the construction of the information noise contrast estimation loss function in subsequent S304.

[0100] S303. Pass the data-augmented image features and through the image feature hashing network H v to obtain two data-augmented image hash codes and and pass the unique text feature E i through the initialized text feature hashing network H e to obtain the text hash code h E ;

[0101] S304. Calculate the information noise contrast estimation loss function L and between the two data-augmented image hash codes, between the data-augmented image hash code and the text hash code h E , and between the data-augmented image hash code and the text hash code respectively, so as to train the image feature hashing network and the text feature hashing network simultaneously; info The expression of the loss function L

[0102] is as follows: info The expression of the loss function L

[0103]

[0104] Among them, L info represents the loss function of information noise contrast estimation InfoNCE, represents the information noise contrast estimation loss function between the two data-augmented image hash codes and , Represents a data-augmented image hash code and the text hash code h E The information noise contrast estimation loss function between Represents another data augmentation image hash code and the text hash code h E The information noise contrast estimation loss function between N and N represents the total number of image samples in the training set, i represents the counting index from 1 to N, and k represents the image hash code. and text hash code It belongs to both positive and negative samples, j represents the image hash code and text hash code belongs to the positive sample (denoted as j∈+), τ2 represents the temperature parameter, and Both represent image features I i The corresponding data-augmented image hash code, and Both represent text features E i The corresponding text hash code.

[0105] S305, using information entropy theory, by converting the hash code in the interval [-1, 1] into a probability value in the interval [0, 1], and calculating the balance loss function L based on the probability value bal , further trained image feature hashing network.

[0106] In this embodiment, information entropy is a quantitative indicator for measuring the uncertainty of random events. When the entropy value is larger, the corresponding distribution tends to be more uniform. In order to ensure that the sum of each hash code in a large number of random samples is close to a uniform distribution, the present invention uses information entropy theory to convert the hash code in the interval [-1, 1] into a probability value in the interval [0, 1], and calculates the balance loss function L based on the probability value. bal , so that the generated hash code can show the characteristics of uniform distribution as much as possible.

[0107] The balance loss function L bal The expression is as follows:

[0108]

[0109] Among them, bit represents the total number of bits of the hash code, [b] represents the bth bit in the total number of bits of the hash code, and Represents hash codes respectively Hash code and hash code The bth position of Represents all image hash codes The hash code obtained after bitwise averaging operation represents a data - enhanced image feature The extracted image hash code represents all image hash codes The hash code obtained after bitwise averaging operation represents another data - enhanced image feature The extracted image hash code represents all text hash codes in the training set The hash code obtained after bitwise averaging operation represents picture feature I i The corresponding unique text feature E i The extracted text hash code, where N represents the total number of image samples in the training set, and i represents the counting index from 1 to N.

[0110] S306. Repeat steps S303 to S305 until the image feature hashing network converges, thus completing the training of the image feature hashing network.

[0111] In this embodiment, the image feature hashing network can fully learn the features of the data through multiple repetitions of the training steps. In each repetition of the training steps, the image feature hashing network updates the weights through backpropagation, gradually reducing the output of the loss function, thereby improving the accuracy and generalization ability of the model.

[0112] S4. Based on the trained image feature hashing network, extract the binary hash codes of the query and the images to be retrieved, and evaluate the retrieval accuracy of the image data by calculating and sorting the Hamming distances between the hash codes, thus completing the hash retrieval of unsupervised depth images. The implementation method is as follows:

[0113] S401. Use the image feature extraction model M L , and extract the image features I q in the query data set X r and the retrieval data set X q respectively and one - to - one correspondence. r ;

[0114] S402. Input the image feature I q and the image feature I r into the trained image feature hashing network H v respectively, and obtain the query image hash code h q and the retrieval image hash code h r in one - to - one correspondence;

[0115] S403. Use the Hamming distance to calculate the query image hash code h q and the retrieval image hash code hr The Hamming distance is calculated, and the results are sorted in ascending order based on the magnitude of the Hamming distance. The top K sorted results are used as the retrieval results. Meanwhile, the MAP@K value calculated using the sorted results is used as the retrieval accuracy of the image data, thus completing the hashing retrieval of unsupervised depth images. Here, MAP@K represents the average of the average accuracies of the top K retrieval results for all queries and is used to measure the sorting quality of the relevant retrieval results.

[0116] Hamming distance d H (h q , h r ) is expressed as follows:

[0117]

[0118] Among them, d H (h q , h r ) represents the Hamming distance between the query image hash code h q and the retrieved image hash code h r . [b] represents the b-th bit in the total number of bits of the hash code, bit represents the total number of bits of the hash code, h q [b] and h r [b] respectively represent the b-th bit of the query image hash code h q and the retrieved image hash code h r . represents the indicator function, which returns 1 when the b-th bits of the query image hash code h q and the retrieved image hash code h r are not equal, otherwise, it returns 0.

[0119] The expression of the MAP@K value is as follows:

[0120]

[0121] Among them, |Q| represents the total number of queries, q represents one query among all queries, AP(q) represents the average accuracy of the top K retrieval results of a query q, r q represents the total number of relevant items, represents the indicator function, which returns 1 when the item in the k-th position is relevant, otherwise, it returns 0. P(k) represents the ratio of the number of relevant contents to the total number in the top K retrieval results, truepositives@K represents the number of relevant contents in the top K retrieval results, and falsepositives@K represents the number of non-relevant contents in the top K retrieval results.

[0122] In summary, the present invention utilizes rich and easily obtainable noun data, and through a special method for constructing external knowledge, generates text features (external knowledge) with higher reliability and discriminative power, which are combined with depth image features to train a hash network for paired graphic-text features, thereby capturing more similarity information. In addition, the present invention can construct supervision information for positive and negative sample pairs by means of the similarity information between external knowledge, guide the contrastive learning strategy, and generate more discriminative hash codes.

Claims

1. An unsupervised deep image hashing retrieval method based on external knowledge, characterized in that It includes the following steps: S1. Obtain an image data set including multiple object categories, and collect a noun data set covering the vocabulary of daily English nouns; S2. Based on the unlabeled image data set and the noun data set, extract image features and noun features respectively, and use the noun features to generate a corresponding unique text feature for each image. Among them, the unique text feature is external knowledge, including the following steps: S201. Construct and initialize a vision-language pre-trained model using pre-trained parameters to obtain an image feature extraction model and a text feature extraction model ; S202. For the pre - processed training data set in the image data and two different data - augmented views of each image data and , use the image feature extraction model to extract features, obtaining the image features of the training set and the corresponding data - augmented image features and ; S203. For the noun dataset , generate multiple corresponding texts for each noun data, and pass the generated texts through the text feature extraction model and calculate the average value to obtain the text feature corresponding to each noun data ; S204. Calculate each text feature Calculate the L2 distance between them, and delete redundant text features with an L2 distance of 0; S205. Use the text feature with the largest distance L2 obtained by screening as the set , and use the clustering algorithm to cluster the text features except for deleting redundancy; S206. Use the text feature that is farthest from the clustering center among the filtered ones as the set , and use the text feature that is closest to the clustering center among the filtered ones as the set ; S207. Based on the set , the set and the set , obtain the text feature set after screening out redundancy ; S208. Sum the text features in the text feature set with the image features of the training set after weighted summation processing to obtain the unique text feature corresponding to each image feature , where the unique text feature is the unique text feature corresponding to the data-augmented image feature and ; ​ S3. Based on the paired image features and text features, respectively construct and train an image feature hashing network and a text feature hashing network, and optimize the image feature hashing network under the guidance of the text feature hash code generated by the text feature hashing network to obtain the trained image feature hashing network. Among them, using the following formula, calculate the cosine similarity between the unique text features : Among them, and respectively represent the unique text features corresponding to different image features, represents two text features and the cosine similarity between them, represents the cosine similarity calculation; Images corresponding to text features with cosine similarity greater than a preset threshold are used as positive samples, and images corresponding to text features less than or equal to the preset threshold are used as negative samples. Among them, the obtained positive and negative samples will be used to guide the construction of the information noise contrast estimation loss function The construction of the loss function The expression of the loss function is as follows: Among them, represents the loss function of InfoNCE (Information Noise Contrastive Estimation), represents the information noise contrastive estimation loss function between two data-augmented image hash codes and ; represents the information noise contrastive estimation loss function between a data-augmented image hash code and a text hash code ; represents the information noise contrastive estimation loss function between another data-augmented image hash code and a text hash code ; represents the total number of image samples in the training set, represents the counting index from 1 to ; represents that the image hash code and the text hash code belong to both positive samples and negative samples, represents that the image hash code and the text hash code belong to positive samples, represents the temperature parameter, , , and all represent the data-augmented image hash codes corresponding to the picture feature ; and all represent the text hash codes corresponding to the text feature ; S4. Based on the trained image feature hashing network, extract the binary hash codes of the query and the image data to be retrieved, and evaluate the retrieval accuracy of the image data by calculating and sorting the Hamming distances between the hash codes, so as to complete the hashing retrieval of the unsupervised depth image.

2. The unsupervised depth image hashing retrieval method based on external knowledge according to claim 1, wherein The S1 includes the following steps: S101. Obtain a number of natural image datasets with accurate labels, and obtain a noun dataset covering the vocabulary of daily English nouns , where the natural image datasets include various objects in life scenes; S102. Split the natural image dataset into a training set , a query set and a retrieval set , where the label information of the natural images in the training set is removed, and the natural images in the query set and the retrieval set retain the label information; S103. For the training set , generate two different data augmentation views through random scaling, cropping, flipping, and color transformation and ; S104. Respectively, the original images of the training set , the query set and the retrieval set are processed by unified scaling and cropping operations to obtain the preprocessed training data set , the query data set and the retrieval data set . At the same time, the label information in the training data set is discarded to obtain an unlabeled training data set, completing the acquisition of an image data set containing multiple object categories. Among them, the image data in the training data set , the query data set and the retrieval data set are of the same dimension as the two data augmentation views and .

3. The unsupervised depth image hashing retrieval method based on external knowledge according to claim 1, characterized in that The unique text feature has the following expression: Among them, represents each text feature with respect to each image feature weight, represents the cosine similarity calculation, represents the temperature parameter, represents the total number of text features after removing redundancy, represents the text feature set all text features in 4. The unsupervised deep image hashing retrieval method based on external knowledge according to claim 1, wherein The S3 includes the following steps: S301. Build and initialize an image feature hashing network and a text feature hashing network respectively wherein, the image feature hashing network and the text feature hashing network both include two fully-connected layers and one hyperbolic tangent (tanh) non-linear layer; ​ S302. Enhance the image features of the data and through the image feature hashing network to obtain two data-enhanced image hash codes and , and enhance the unique text features through the initialized text feature hashing network to obtain the text hash code ; S303. Calculate the image hash codes of the two data-augmented images respectively and the information noise contrast estimation loss function between the data-augmented image hash code and the text hash code and the text hash code to train the image feature hash network and the text feature hash network simultaneously; ​ S304. Using the information entropy theory, convert the hash codes within the interval into probability values within the interval, and calculate the balanced loss function based on the probability values to further train the image feature hashing network; S305. Repeat steps S302 to S304 until the image feature hashing network converges, so as to complete the training of the image feature hashing network.

5. The unsupervised depth image hashing retrieval method based on external knowledge according to claim 4, wherein The balanced loss function has the following expression: Among them, represents the total number of bits of the hash code, represents the th bit in the total number of bits of the hash code, , and respectively represent the th bit of the hash code , the hash code th bit of and the hash code represents the hash code obtained after bitwise averaging of all image hash codes , represents an image hash code extracted from a data - enhanced image feature , represents the hash code obtained after bitwise averaging of all image hash codes , represents an image hash code extracted from another data - enhanced image feature , represents the hash code obtained after bitwise averaging of all text hash codes in the training set, represents the unique text feature corresponding to the picture feature extracted text hash code, represents the total number of image samples in the training set, represents the counting index from 1 to and​ 6. The unsupervised depth image hashing retrieval method based on external knowledge according to claim 4, characterized in that, The S4 includes the following steps: S401. Use the image feature extraction model to extract the image features in the query dataset and the retrieval dataset one by one respectively and the image features ; S402. Input the image feature and the image feature into the pre-trained image feature hashing network respectively, and obtain the query image hash code and the retrieved image hash code in one-to-one correspondence; S403. Calculate the hash code of the query image using the Hamming distance and the hash code of the retrieved image to calculate the Hamming distance, and sort them in ascending order based on the magnitude of the Hamming distance. Take the first sorting results as the retrieval results. At the same time, use the value calculated from the sorting results as the retrieval accuracy of the image data to complete the hash retrieval of the unsupervised depth image. Among them, represents the average of the average accuracies of the first retrieval results of all queries, which is used to measure the sorting quality of the relevant retrieval results.

7. The unsupervised deep image hashing retrieval method based on external knowledge according to claim 6, characterized in that The Hamming distance is expressed as follows: Among them, represents the Hamming distance between the query image hash code and the retrieved image hash code, represents the th bit in the total number of bits of the hash code, represents the total number of bits of the hash code, and respectively represent the th bit of the query image hash code and the retrieved image hash code ; represents the indicator function, and returns 1 when the th bit of the query image hash code and the retrieved image hash code are not equal, otherwise, returns 0.

8. The unsupervised depth image hashing retrieval method based on external knowledge according to claim 7, characterized in that The said The expression of the value is as follows: Among them, represents the total number of queries, represents a query among all queries, represents a query the average precision of the top represents the total number of relevant items, represents the indicator function, which returns 1 when the item at the th position is relevant, otherwise, returns 0, the ratio of the number of relevant contents to the total number among the top the K number of relevant contents among the top the K number of non - relevant contents among the top

Images