Weakly supervised hashing image retrieval method and system based on large-scale visual language model

Abstract

The application discloses a weakly supervised hash image retrieval method and system based on a large-scale visual language model. A large-scale visual language model CLIP is fine-tuned using a training set containing weakly supervised labels, and linear interpolation is performed on the obtained fine-tuning weights and original weights to obtain model weights for subsequent use. A weakly supervised label optimization module is designed to eliminate redundant features in the inference stage. A pre-trained AlexNet network is used to extract visual features of images, and the image features are mapped into hash codes and feature vectors with the same dimension as the text features through a hash layer and a reconstruction layer. A multi-head self-attention mechanism is used to calculate the scores of each label corresponding to the picture. A balanced loss is designed to make the model pay more attention to the tail classes during the training process. The application can effectively alleviate the long-tail problem of sample distribution in the data set and more accurately realize weakly supervised hash image retrieval.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of weakly supervised hash image retrieval, in particular to a weakly supervised hash image retrieval method and system based on a large-scale visual language model. BACKGROUND

[0002] In today's information explosion era, image as an important medium for information transmission, its retrieval efficiency and accuracy have been widely concerned. Although supervised hash image retrieval task can achieve good results after training with a large amount of manually labeled data. However, these methods usually require a large amount of labeled data to train the model, which not only increases the labor cost, but also limits the application range of the model. For some unsupervised deep hash methods, although the problem of time-consuming and labor-intensive labeling of massive data in supervised deep hash is solved, but because there is no visible labeled data for hash learning, the learned hash code cannot well represent the image content.

[0003] With the rapid development of the Internet, many pictures on social media sites contain user-provided tags. These user-provided tags can represent the semantic information of the image to some extent, thereby reducing the dependence on a large amount of labeled data. However, since these tags are not labeled for the purpose of training a supervised model, they usually contain information unrelated to the content of the picture, and the categories of these tags present a clear long-tail problem. Therefore, we can only regard these user-provided tags as weakly supervised information.

[0004] At present, many methods for weakly supervised hash image retrieval have been proposed. Gattupalli et al. proposed the WDHT model, which uses the average aggregated word2vec label vector as weakly supervised input; Jin et al. proposed the MGRN model, which predicts random mask labels by training a graph model to learn joint visual semantic representation in a self-supervised manner; Wang et al. proposed the EWDH method, which optimizes user-provided tags based on image content to obtain more accurate weakly supervised signals. Du et al. proposed the WSHRCA, which updates the hash code by reconstructing the image vector and further optimizes it through cross-modal attention mechanism.

[0005] Although the above models have achieved good results, there are still deficiencies, for example, the WDHT model equally uses each weakly supervised label and cannot solve the noise problem in the label. The MGRN model only captures the relationship between the image and the label in the original label set and does not optimize the composition of the label set. The EWDH method optimizes the label, but the similarity calculation used does not have good cross-modal understanding ability. The WSHRCA model does not filter the labels more related to the content of the picture before calculating the attention score. At the same time, the above methods also do not pay attention to the performance decline caused by the tail classes in the long-tail distribution. SUMMARY

[0006] The application discloses a weakly supervised hash image retrieval method and system based on a large-scale visual language model, which can effectively improve the performance of weakly supervised hash image retrieval.

[0007] The technical solution of the application is as follows: a weakly supervised hash image retrieval method based on a large-scale visual language model, comprising:

[0008] Step 1: fine-tune the contrast learning language-image pre-training model CLIP using the training set containing weakly supervised labels, the training set containing natural pictures of different categories and corresponding weakly supervised labels provided by users, and the original pre-training weight and the fine-tuned weight are fused by a linear interpolation method;

[0009] Step 2: design a weakly supervised label optimization module to replace the noise label with a label more related to the image content;

[0010] Step 3: use a pre-trained convolutional neural network to extract image features, and then train a hash network to map the image features into feature vectors with the same dimension as the hash code and the text feature;

[0011] Step 4: use a multi-head self-attention mechanism to interact the image features and the text features to dynamically adjust the distribution of attention;

[0012] Step 5: design a balanced loss to further constrain the parameter update in the hash network;

[0013] Step 6: obtain the hash code of the query picture and the database picture through the trained hash network model, calculate the Hamming distance between the hash codes and sort them, and finally select the picture set with the smallest distance as the hash image retrieval result. Further, the first step uses the training data S train ={(x i ,T i )|i=1,…,N tr}fine-tune the CLIP model, wherein x i ∈R H×W×3 representing images in the training set, T i representing image x i corresponding weakly supervised label set, N tr denotes the total number of images in the training set; use all images in the training set as visual input for the fine-tuning process, for image x i use prompt template a photo of T i as its corresponding text description, where T i each label in the label set is connected by a space into a string; cross-entropy loss is used for parameter optimization in both image and text ends in the fine-tuning process, and the fine-tuned weight is denoted as θ finetune , the original weight is denoted as θ zeroshot ; linear interpolation is performed between the original model weight and the fine-tuned model weight to obtain a new model weight denoted as θ final :

[0014] θ final = (1-α)·θ zeroshot +α·θ finetune

[0015] wherein, the interpolation coefficient α is used to control the contribution proportion of the model weight.

[0016] Further, the second step designs a weakly supervised label optimization module to optimize the noisy label;

[0017] 2.1 In the process of optimizing the original weakly supervised label, first, the image features and text features obtained by the model weight θ final are multiplied element by element after being normalized by the Euclidean norm, to obtain the original cross-modal cross-representation F o , which is calculated as follows:

[0018]

[0019] wherein F i represents the image features generated by the CLIP image encoder after using the model weight θ final , F t represents the text features generated by the CLIP text encoder after using the model weight θ final ;

[0020] Then, the similarity score s is calculated using the class label feature F c and the text feature F t as follows:

[0021]

[0022] The ratio of the similarity score of each class to the average score is used as the weight w, which is calculated as follows:

[0023]

[0024] After that, each feature is weighted by category, and then the mean value is calculated in the category dimension as the redundant feature F r , described by the following formula:

[0025] F r = mean(F o ☉expand(w))

[0026] By removing the redundant feature F r from the original cross-modal representation F o , a new cross-modal representation F n is obtained, calculated as follows:

[0027] F n = F o -expand(F r ) Assuming that the number of common labels in the weakly supervised label set T i corresponding to the image x i is α, then the top α labels with the highest relevance to the image content are taken as the optimized labels, calculated as follows:

[0028]

[0029] 2.2 Optimize the weakly supervised labels of all pictures in the training set S train according to the process in 2.1 to obtain the optimized label set T * that filters out noise labels and reduces the impact of redundant features, and use the CLIP text encoder after the model weight θ final to convert the labels in T * into text feature vectors t.

[0030] Further, the third step uses a pre-trained convolutional neural network to extract image features; a fully connected layer in the hash network maps the dimension of the image features to the dimension corresponding to the number of bits of the hash code, and uses the Sigmoid function for activation; this process uses quantization loss L1 for parameter optimization, as follows:

[0031]

[0032] where m represents the size of the minimum batch, b represents the number of bits in the hash code, and h idenotes the hash code, I is an all-one vector with the same dimension as the hash code; the obtained hash feature vector h is then passed through an fully connected layer again and activated using a LeakyReLU function, and the h feature dimension is mapped to a vector v with the same dimension as t, so as to realize cross-modal feature interaction later.

[0033] Further, the fourth step uses a multi-head self-attention mechanism to interact the image feature v obtained in the third step and the text feature t obtained in the second step; for each head There are three weight matrices for calculating Query, Key and Value, satisfying: The attention weight is calculated using a scaled dot-product attention mechanism, and the formula is as follows:

[0034]

[0035] where d k represents the dimension of each head, and the obtained attention output is denoted as:

[0036]

[0037] The outputs of all heads are spliced together, and then a linear layer is used for calculation, as follows:

[0038] Z=Concat(Z1,Z2,…,Z h )W O

[0039] where W O is the weight matrix of the linear layer; then a feed-forward network is used to further process and convert the features of the attention layer output, which includes two fully connected layers, an activation layer and a Dropout layer; the first fully connected layer changes the feature dimension from D to 2xD, and the second fully connected layer restores the feature dimension from 2xD to D, and the activation layer in the middle uses a ReLU activation function, and the calculation is as follows:

[0040] FFN(Z)=W2max(0,W1Z+b1)+b2

[0041] where w1, b1, W2, and b2 are parameters of the two fully connected layers; the multi-head attention output is connected in residual with the output of the feed-forward network to obtain the final text vector t * , and the calculation is as follows:

[0042] t * =Z+FFN(Z)

[0043] The dynamic and adaptive text vector t *Aligning with the corresponding image feature v, this process uses hinge loss L2 to optimize the parameters, as follows:

[0044]

[0045] where ε represents the boundary parameter; the pairwise similarity loss L3 is used to achieve a similar distribution of Hamming distance and cosine distance in the hash space, as follows:

[0046]

[0047] Further, the fifth step designs a balance loss to further constrain the hash network model parameter optimization after the quantization loss L1, hinge loss L2 and pairwise similarity loss L3, and the balance loss L4 is calculated as follows:

[0048]

[0049] where the parameter β is the balance factor, C is the new label set T * generated after the weakly supervised label optimization module, represents the picture x i in T * , t c is the one-hot encoding in T * , t final represents the text vector generated after the CLIP text encoder using the weight θ q , and γ is the adjustment factor for adjusting the attention degree to the more difficult samples; the balance loss L4 can better learn the label samples with low similarity by dynamically adjusting the classification weight.

[0050] Further, the sixth step obtains the hash code of the query picture and the database picture through the trained network model, calculates the Hamming distance between the hash codes and sorts them, and finally selects the picture set with the smallest distance as the hash image retrieval result; where the hash code of the query picture generated by the model is h d , and the hash code of the database picture is h d ; the Hamming distance between the two is calculated as follows:

[0051]

[0052] where b represents the number of bits of the hash code, h q [k] and h d [k] are the kth bit of the hash codes h q and h d , respectively; according to the calculated Hamming distance, all the database pictures are sorted, and the first N pictures with the smallest distance are selected as the final weakly supervised hash image retrieval result.

[0053] A weakly supervised hash image retrieval system based on a large-scale visual language model, comprising:

[0054] A fine-tuning module fine-tunes a contrast learning language image pre-training model CLIP using a training set containing weakly supervised labels, and fuses the original pre-training weight and the fine-tuned weight through a linear interpolation method;

[0055] A weakly supervised label optimization module is used to replace the noise label with a label having higher relevance to the image content;

[0056] A feature extraction module extracts image features using a pre-trained convolutional neural network, and maps the image features into feature vectors of the same dimension as the hash code and the text feature;

[0057] A feature interaction module uses a multi-head self-attention mechanism to interact the image features and the text features, and dynamically adjusts the distribution of attention;

[0058] A loss function module is used to further constrain the parameter update in the hash network;

[0059] A hash image retrieval module obtains the hash code of the query picture and the database picture through the trained hash network model, calculates the Hamming distance between the hash codes and performs sorting, and finally selects the picture set with the smallest distance as the hash image retrieval result.

[0060] Compared with the prior art, the present application has the following significant features: (1) a weakly supervised label optimization module is designed, which uses the good cross-modal alignment capability of the large-scale visual language model CLIP to optimize the noise label, and solves the problems of the influence of the fine-tuning process on the long-tail distribution of downstream data and the mutual influence between classes in the reasoning stage; (2) a new balance function is designed, so that the model can pay more attention to the learning of tail samples during the training process. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a flowchart of a weakly supervised hash image retrieval method based on a large-scale visual language model of the present application.

[0062] Figure 2 is a network structure diagram of the present application.

[0063] Figure 3 is a result diagram of the weakly supervised label optimization in the present application.

[0064] Figure 4 is a structure diagram of a weakly supervised hash image retrieval system based on a large-scale visual language model of the present application. DETAILED DESCRIPTION

[0065] The application fine-tunes the CLIP model using the data of the training set, and realizes the fusion of the weights through linear interpolation. A weakly supervised label optimization module is designed, which reasons the labels before using multi-head self-attention to obtain a new label set with higher relevance to the content of the picture. Finally, a balanced loss is designed, so that the model can pay more attention to the learning of the tail samples in the training process, and high-precision hash image retrieval is realized.

[0066] The application will be further described below in conjunction with the drawings of the specification

[0067] In conjunction with Figure 1 , the steps of the application of weakly supervised hash image retrieval based on a large-scale visual language model are described in detail.

[0068] The first step uses training data S train = {(x i , T i )|i = 1,..., N tr} to fine-tune the CLIP model, where x i ∈R H ×W×3 represents the images in the training set, T i represents the corresponding weakly supervised label set of the image x i , and N tr represents the total number of pictures in the training set; the network structure of the application is shown in Figure 2 , all pictures in the training set are used as visual input in the fine-tuning process, and the prompt template a photo of T i is used as the corresponding text description for the image x i , where each label in the T i label set is connected by a space to form a string; the cross-entropy loss is used for parameter optimization in the fine-tuning process for both the image end and the text end. The fine-tuned weight is denoted as θ finetune , and the original weight is denoted as θ zeroshot ; the original model weight and the fine-tuned model weight are linearly interpolated to obtain a new model weight denoted as θ final :

[0069] θ final = (1-α)·θ zeroshot +α·θ finetune

[0070] Where the interpolation coefficient α can control the contribution proportion of the model weight, which is usually set to 0.5. Through linear interpolation between the weights, the influence of the model parameters on the uneven distribution of downstream data in the fine-tuning process can be reduced.

[0071] The second step is to design a weakly supervised label optimization module to optimize the noisy labels and reduce the mutual influence of redundant features between categories.

[0072] 2.1 In the process of optimizing the original weakly supervised labels, the normalized Euclidean norm is first used to normalize the model weight θ final The obtained image features and text features are multiplied element by element to obtain the original cross-modal cross-representation F o The calculation is as follows:

[0073]

[0074] Where F i represents the image features generated by the CLIP image encoder after using the model weight θ final F t represents the text features generated by the CLIP text encoder after using the model weight θ final Then, the similarity score s is calculated using the category label feature F c and the text feature F t as follows:

[0075]

[0076] The ratio of the similarity score of each category to the average score is used as the weight w, which is calculated as follows:

[0077]

[0078] After that, each feature is weighted by the category, and then the average is calculated in the category dimension as the redundant feature F r , which can be described in the following form:

[0079] F r = mean(F o ☉expand(w))

[0080] By removing the redundant feature F o from the original cross-modal cross-representation F r , a new cross-modal cross-representation F n can be obtained, which is calculated as follows:

[0081] F n = F o -expand(F r )

[0082] Eliminating redundant features can reduce the mutual influence between categories during model prediction, resulting in more accurate category rankings. After that, assuming that the image x i corresponds to the weakly supervised label set T iThe number of common tags in the image is α, and the top α tags with the highest relevance to the image content in the reasoning stage are selected as the optimized tags, which are calculated as follows:

[0083]

[0084] 2.2 Optimize the training set S according to the process of 2.1 train The weakly supervised labels of all pictures in the image are filtered out, as shown in Figure 3 , to obtain the optimized label set T * , which filters out noise labels and reduces the influence of redundant features. final The CLIP text encoder after the model weight θ * converts the labels in T i into 768-dimensional text feature vectors t.

[0085] The third step uses the pre-trained AlexNet network to extract 4096-dimensional image features; in the hash network, a fully connected layer is used to map the dimension of the image features to the dimension corresponding to the number of bits of the hash code, and the Sigmoid function is used for activation; this process uses the quantization loss L1 for parameter optimization, and the formula is as follows:

[0086]

[0087] where m represents the size of the minimum batch, b represents the number of bits in the hash code, h i represents the hash code, and I is a full 1 vector with the same dimension as the hash code; then the obtained hash feature vector h is passed through a fully connected layer again, and the LeakyReLU function is used for activation, which maps the h feature dimension to a vector v with the same dimension as t, realizing the cross-modal feature interaction.

[0088] The fourth step uses the multi-head self-attention mechanism to interact the image features v obtained in the third step and the text features t obtained in the second step; for each head , there are three weight matrices for calculating Query, Key and Value, which satisfy: The scaled dot-product attention mechanism is used to calculate the attention weight, and the formula is as follows:

[0089]

[0090] where d k represents the dimension of each head, and the calculated attention output is represented as:

[0091]

[0092] The outputs of all heads are concatenated together, and then a linear layer is used for calculation, as follows:

[0093] Z = Concat(Z1, Z2, …, Z h )W O

[0094] where W O is the weight matrix of the linear layer; then a feed-forward network is used to further process and transform the features output by the attention layer, which contains two fully connected layers, an activation layer and a Dropout layer; the first fully connected layer changes the feature dimension from D to 2xD, and the second fully connected layer restores the feature dimension from 2xD to D, and the activation layer in the middle uses the ReLU activation function, which is calculated as follows:

[0095] FFN(Z) = W2max(0, W1Z + b1) + b2

[0096] where W1, b1, W2, b2 are the parameters of the two fully connected layers; the output of the multi-head attention is used in residual connection with the output of the feed-forward network to obtain the final text vector t * , which is calculated as follows:

[0097] t * = Z + FFN(Z)

[0098] The dynamic adaptive text vector t * is aligned with the corresponding image feature v, and this process uses hinge loss L2 for parameter optimization, which is calculated as follows:

[0099]

[0100] where ε is set to 0.7 as a boundary parameter; pairwise similarity loss L3 is used to achieve similar distribution of Hamming distance and cosine distance in the hash space, which is calculated as follows:

[0101]

[0102] A balance loss is designed in the fifth step to further constrain the model after losses L1, L2 and L3, so that the model pays more attention to difficult samples in the presence of long-tail distribution in the training set, and the balance loss L4 is calculated as follows:

[0103]

[0104] where the parameter β is the balance factor, C is the number of classes in the new label set T * generated after the weakly supervised label optimization module, represents the one-hot encoding of picture x i in T * , and t c represents the one-hot encoding of picture x *The C-th text input is input to use θ final The text vector generated after the CLIP text encoder of the weight, γ is used as an adjustment factor, is used to adjust the attention degree to the more difficult sample; the balanced loss L4 is realized by dynamically adjusting the classification weight, and the label sample with lower similarity can be better learned.

[0105] The sixth step is to obtain the hash code of the query picture and the database picture through the trained network model, calculate the Hamming distance between the hash codes and sort, and finally select the picture set with the smallest distance as the hash image retrieval result; wherein, assuming that the hash code of the query picture generated by the model is h q , and the hash code of the database picture is h d ; then, the Hamming distance between the two is calculated as follows:

[0106]

[0107] Wherein, b represents the number of bits of the hash code, h q [k] and h d [k] are the kth bit of the hash code h q and h d ; according to the calculated Hamming distance, all database pictures are sorted, and the first N pictures with the smallest distance are selected as the final weakly supervised hash image retrieval result.

[0108] In combination Figure 4 , the weakly supervised hash image retrieval system based on a large-scale visual language model comprises:

[0109] The fine-tuning module is used for fine-tuning the language image pre-training model CLIP of contrast learning by using the training set containing weakly supervised labels, and fusing the original pre-training weight and the fine-tuned weight by using the linear interpolation method.

[0110] The weakly supervised label optimization module is used for replacing the noise label with a label with higher relevance to the image content.

[0111] The feature extraction module adopts a pre-trained convolutional neural network to extract image features, and maps the image features into feature vectors with the same dimension as the hash code and the text feature.

[0112] The feature interaction module uses a multi-head self-attention mechanism to interact the image features and the text features, and dynamically adjusts the distribution of attention.

[0113] The loss function module is used for further constraining the parameter update in the hash network.

[0114] The hash image retrieval module obtains hash codes of the query picture and the database picture through the trained hash network model, calculates the Hamming distance between the hash codes and performs sorting, and finally selects a picture set with the smallest distance as the hash image retrieval result.

[0115] Experimental results

[0116] The performance of the model on hash image retrieval is verified by the mAP value, and the larger the mAP value is, the better the performance of the model is. The weakly supervised hash image retrieval method based on the large-scale visual language model is compared with the current advanced multi-label classification method in the world, and the comparison methods include WDHT, MGRN, EWSH and WSHRCA methods.

[0117] Tables 1 and 2 are the mAP values calculated by using all the returned results of the results of the application and the above four models in the MirFlicker and NUS-WIDE two data sets. Among them, the MirFlicker data set contains 38 semantic labels for testing and 1386 common weakly supervised labels provided by users, from which 15240 pictures are randomly taken as the training set and the data set, and 2000 pictures are taken as the query set. In the NUS-WIDE data set, the most common 10 categories are used for testing, and the 638 common weakly supervised labels are used for training, from which 15240 pictures are randomly taken as the training set, 2000 pictures are taken as the query set, and the remaining pictures are taken as the data set.

[0118] Table 1 Comparison of MirFlicker experimental results

[0119]

[0120] Note: Bold font represents the optimal result of each row

[0121] Table 2 Comparison of NUS-WIDE experimental results

[0122]

[0123] Note: Bold font represents the optimal result of each row

[0124] From the comparison results of Tables 1 and 2, it can be seen that the retrieval performance of the application is improved in the MirFlicker and NUS-WIDE data sets under different bit numbers.

Claims

1. A weakly supervised hash image retrieval method based on a large-scale visual language model, characterized in that, The method comprises the following steps: Step 1: fine-tune the language-image pre-training model CLIP of contrast learning by using a training set containing weakly supervised labels, and fuse the original pre-training weight and the fine-tuned weight by linear interpolation; Step 2: design a weakly supervised label optimization module for replacing the noise label with a label having higher relevance to the image content; Step 3: extract image features by using a pre-trained convolutional neural network, and then train a hash network to map the image features into feature vectors of the same dimension as hash codes and text features; Step 4: use a multi-head self-attention mechanism to interact the image features and the text features, and dynamically adjust the distribution of attention; Step 5: design a balance loss for further constraining the parameter update in the hash network; Step 6: obtain the hash codes of the query picture and the database picture by using the trained hash network model, calculate the Hamming distance between the hash codes, and sort the hash codes, and finally select the picture set with the smallest distance as the hash image retrieval result; The first step uses training data S train = {(x i , T i )|i = 1, …, N tr} fine-tunes the CLIP model, where x i ∈ R H×W×3 represents an image in the training set, T i represents the corresponding weakly supervised label set of the image x i , and N tr represents the total number of pictures in the training set; all pictures in the training set are used as visual input in the fine-tuning process, and the prompt template a photo ofT i is used as the corresponding text description for the image x i , where each label in the label set T i is connected by a space into a string; both the image end and the text end in the fine-tuning process use cross-entropy loss to optimize the parameters, and the fine-tuned weight is denoted as θ finetune , the original weight is denoted as θ zeroshot ; linear interpolation is performed on the original model weight and the fine-tuned model weight to obtain a new model weight denoted as θ final : θ final = (1 - a) · θ zeroshot + a · θ finetune The interpolation coefficient alpha is used to control the contribution proportion of the model weight; The weakly supervised label optimization module is designed in step 2 to optimize the noise label; 2.1 In the process of optimizing the original weakly supervised label, the normalized image feature and text feature obtained by the model weight θ final are multiplied element by element to obtain the original cross-modal cross representation F o , which is calculated as follows: where F i representing using model weights θ final image features generated by the post-CLIP image encoder, F t representing using model weights θ final text features generated by the post-CLIP text encoder; Then the category label feature F is used c and the text feature F t The similarity score s is computed as follows: The ratio of the similarity score of each category to the average score is used as the weight w and calculated as follows: After that, each feature is weighted by category, and then the mean is taken in the category dimension as the redundant feature F r , which is described by the following formula: F r = mean(F o ⊙expand(w)) By removing the redundant features F o redundant features F r , the new cross-modal cross-representation F n is calculated as follows: F n = F o expand(F r ) Assume image x i The corresponding weakly supervised label set T i The number of common labels in T α Take the top 2.2 Optimize training set S following the procedure in 2.1 train with weakly supervised labels of all images, resulting in an optimized label set T that filters out noisy labels and reduces the impact of redundant features * , using the model weights final The CLIP text encoder after * converts the labels in T into text feature vectors t; In step 3, the image features are extracted by using a pre-trained convolutional neural network; in the hash network, the dimension of the image features is mapped to the dimension corresponding to the bit number of the hash code by using a fully connected layer, and the Sigmoid function is used for activation; this process uses a quantization loss L1 for parameter optimization, and the formula is as follows: wherein m denotes the size of the minimum batch, b represents the number of bits in the hash code, h i h represents the hash code, and I is a full-1 vector with the same dimension as the hash code; the obtained hash feature vector h is then passed through a fully connected layer again and activated using a LeakyReLU function, so as to map the h feature dimension into a vector v with the same dimension as t, so as to realize the cross-modal feature interaction. The fourth step uses a multi-head self-attention mechanism to interact the image feature v obtained in the third step and the text feature t obtained in the second step; for each head There are three weight matrices For calculating Query, Key and Value, meet: The attention weight is calculated using the scaled dot-product attention mechanism, and the formula is as follows: where d k representing the dimension of each head, the computed attention output is represented as: The outputs of all heads are spliced together, and then a linear layer is used for calculation as follows: Z = Concat(Z1, Z2,..., Z h )W O where W O is the weight matrix of the linear layer; then a feed-forward network is used to further process and transform the features of the attention layer output, which contains two fully connected layers, an activation layer and a Dropout layer; the first fully connected layer transforms the feature dimension from D to 2xD, and the second fully connected layer restores the feature dimension from 2xD to D, and the activation layer in the middle adopts the ReLU activation function, which is calculated as follows: FFN(Z)=W2max(0,W1Z+b1)+b2 where W1,b1,W2,b2 are the parameters of the two fully connected layers; the output of the multi-head attention is used to make a residual connection with the output of the feed-forward network to obtain the final text vector t * , which is calculated as follows: t * = Z + FFN(Z) Text vector t using dynamic adaptation * This process employs a hinge loss L2 for parameter optimization, aligned with the corresponding image feature v, as follows: Wherein, epsilon represents a boundary parameter; the pairwise similarity loss L3 is used to realize that the Hamming distance and the cosine distance in the hash space have similar distributions, and the formula is as follows: In step 5, a balance loss is designed to further constrain the parameter optimization of the hash network model after the quantization loss L1, the hinge loss L2 and the pairwise similarity loss L3, and the balance loss L4 is calculated as follows: wherein, the parameter β is a balance factor, C is a new label set T generated after the weakly supervised label optimization module * the number of categories in T represent the picture x i one-hot encoding in T * , t c represents the text vector generated after the Cth text input in T * is input into the CLIP text encoder using θ final weight, γ is used as an adjustment factor to adjust the attention degree to the more difficult samples; the balance loss L4 realizes better learning of label samples with lower similarity by dynamically adjusting the classification weight. The sixth step is to obtain the hash codes of the query picture and the database picture by the trained network model, calculate the Hamming distance between the hash codes and sort them, and finally select the picture set with the smallest distance as the hash image retrieval result; wherein, assuming that the hash code of the query picture generated by the model is h q , and the hash code of the database picture is h d ; the Hamming distance between the two is calculated according to the following formula: where b represents the number of bits of the hash code, h q [k] and h d [k] are the kth bits of hash codes h q and h d respectively; all database images are sorted according to the calculated Hamming distance, and the first N images with the smallest distance are selected as the final weakly supervised hash image retrieval result.

Images