Weak supervision hash image retrieval method and system based on large-scale visual language model
By using large-scale visual language model CLIP in weakly supervised hash image retrieval for fine-tuning and label optimization, combining the multi-head self-attention mechanism and balanced loss function, the problems of noise labels and long-tail distribution are solved, and the retrieval performance is improved, thus realizing the More efficient image retrieval.
Patent Information
- Application Number
- CN202410791031.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-19
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-06-19
AI Technical Summary
The existing weakly supervised hash image retrieval methods have problems with performance degradation when processing noise labels and long-tail distributions, and fail to effectively utilize the tag information provided by users, resulting in limited search efficiency and accuracy.
The large-scale visual language model CLIP is used for fine-tuning, and the weight is fusion through linear interpolation, a weakly supervised label optimization module is designed to replace the noise label, interactive image and text features are used by a multi-head self-attention mechanism, and the hash network is optimized in combination with the balance loss function to improve retrieval performance.
The performance of weakly supervised hash image retrieval is significantly improved, especially when dealing with noise labels and long-tail distributions, and the accuracy and efficiency of the retrieval is improved, allowing better attention to the difficult samples at the tail.
Smart Images

Figure CN119025695A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of weakly supervised hash image retrieval, and specifically, is a weakly supervised hash image retrieval method and system based on a large-scale visual language model. Background Art
[0002] In today's era of information explosion, images are an important medium for information transmission, and their retrieval efficiency and accuracy have received widespread attention. Although supervised hash image retrieval tasks can achieve good results after training with a large amount of manually annotated data. However, these methods usually require a large amount of annotated data to train the model, which not only increases the labor cost but also limits the scope of application of the model. For some unsupervised deep hashing methods, although they solve the problem of time-consuming and labor-intensive labeling of massive data in supervised deep hashing, because there is no visible annotated data for hash learning, the learned hash code cannot represent the image content well.
[0003] With the rapid development of the Internet, many images on social networking sites contain user-provided tags. These user-provided tags can represent the semantic information of the image to a certain extent, thereby reducing the reliance on a large amount of labeled data. However, since these tags are not annotated for the purpose of training supervised models, they usually contain information that is irrelevant to the content of the image, and the categories of these tags show an obvious long-tail problem. Therefore, we can only regard these user-provided tags as weak supervision information.
[0004] At present, many methods for weakly supervised hash image retrieval have been proposed. Gattupalli et al. proposed the WDHT model, which uses the average aggregated word2vec label vector as the weak supervision input; Jin et al. proposed the MGRN model, which learns the joint visual semantic representation in a self-supervised manner by training a graph model to predict random mask labels; Wang et al. proposed the EWDH method, which optimizes the user-provided labels according to the image content to obtain a more accurate weak supervision signal. Du et al. proposed WSHRCA, which explicitly updates the hash code by reconstructing the image vector through the hash code, and further optimizes it through the cross-modal attention mechanism.
[0005] Although the above models have achieved good results, they still have some shortcomings. For example, the WDHT model uses each weakly supervised label equally and cannot solve the noise problem in the label. The MGRN model only captures the relationship between the image and the label in the original label set, and does not optimize the composition of the label set. Although the EWDH method optimizes the label, the similarity calculation used does not have good cross-modal understanding capabilities. The WSHRCA model does not filter labels that are more relevant to the image content before calculating the attention score. At the same time, the above methods have not paid attention to the performance degradation caused by the more difficult categories at the tail of the long-tail distribution. Summary of the invention
[0006] The present invention discloses a weakly supervised hash image retrieval method and system based on a large-scale visual language model, which can effectively improve the weakly supervised hash image retrieval performance.
[0007] The technical solution to realize the present invention is: a weakly supervised hash image retrieval method based on a large-scale visual language model, comprising:
[0008] Step 1: Fine-tune the CLIP model for contrastive learning using a training set containing weakly supervised labels. The training set contains natural images of different categories and their corresponding weakly supervised labels provided by users. The original pre-trained weights and the fine-tuned weights are fused by linear interpolation.
[0009] Step 2: Design a weakly supervised label optimization module to replace noisy labels with labels that are more relevant to the image content;
[0010] Step 3: Use a pre-trained convolutional neural network to extract image features, and then train a hash network to map image features into feature vectors of the same dimensions as the hash code and text features;
[0011] Step 4: Use the multi-head self-attention mechanism to interact image features and text features and dynamically adjust the allocation of attention;
[0012] Step 5: Design a balanced loss to further constrain parameter updates in the hash network;
[0013] Step 6: Obtain the hash codes of the query image and database images through the trained hash network model, calculate the Hamming distance between the hash codes and sort them, and finally select the image set with the smallest distance as the hash image retrieval result. train ={(x i ,T i )|i=1,…,N tr} Fine-tune the CLIP model, where x i ∈R H×W×3 represents the images in the training set, T i Represents image x i The corresponding weakly supervised label set, N tr Represents the total number of images in the training set; use all images in the training set as the visual input of the fine-tuning process. For image x i Use the prompt template a photo of T i As its corresponding text description, where T i Each label in the label set is connected into a string with spaces; in the fine-tuning process, cross entropy loss is used to optimize the parameters of both the image side and the text side, and the weight after fine-tuning is recorded as θ finetune , the original weight is recorded as θ zeroshot ; Linearly interpolate the original model weights with the fine-tuned model weights to obtain a new model weight denoted as θ final :
[0014] θ final =(1-α)·θ zeroshot +α·θ finetune
[0015] Among them, the interpolation coefficient α is used to control the contribution ratio of the model weight.
[0016] Furthermore, in the second step, a weakly supervised label optimization module is designed to optimize the noisy labels;
[0017] 2.1 In the process of optimizing the original weakly supervised labels, we first use the model weights θ normalized by the Euclidean norm final The obtained image features and text features are multiplied element by element to obtain the original cross-modal cross representation F o , calculated as follows:
[0018]
[0019] where F i Represents the weight of the model used final Image features generated by the post-CLIP image encoder, F t Represents the weight of the model used final Text features generated by the post-CLIP text encoder;
[0020] Then use the class label feature F c and text features F t The similarity score s is calculated as follows:
[0021]
[0022] The ratio of the similarity score of each category to the mean score is used as the weight w and is calculated as follows:
[0023]
[0024] Then, each feature is weighted by category, and the average is calculated in the category dimension as the redundant feature F. r , the formula is described as follows:
[0025] F r =mean(F o ☉ expand(w)
[0026] By transforming the original cross-modal representation F o Remove redundant features F r , and obtain a new cross-modal cross representation F n The calculation is as follows:
[0027] F n =F o -expand(F r ) Assume that image x i The corresponding weakly supervised label set T i The number of common labels in is α, then take the top labels with the highest correlation with the image content. α The optimized labels are calculated as follows:
[0028]
[0029] 2.2 Optimize the training set S according to the process in 2.1 train Weakly supervised labels of all images in the image are obtained to obtain the optimized label set T that filters out noise labels and reduces the influence of redundant features. * , using the model weights θ final The CLIP text encoder will then * The labels in are transformed into text feature vectors t.
[0030] Furthermore, in the third step, a pre-trained convolutional neural network is used to extract image features. In the hash network, a fully connected layer is used to map the dimensions of image features into the dimensions corresponding to the number of hash code bits, and a Sigmoid function is used for activation. This process uses the quantization loss L1 for parameter optimization, and the formula is as follows:
[0031]
[0032] Where m is the size of the minimum batch, b is the number of bits in the hash code, and h is irepresents the hash code, Ι is the all-1 vector of the same dimension as the hash code; the obtained hash feature vector h is then passed through a fully connected layer again and activated using the LeakyReLU function to map the h feature dimension to a vector ν of the same dimension as t to achieve subsequent cross-modal feature interaction.
[0033] Furthermore, in the fourth step, the multi-head self-attention mechanism is used to interact the image feature ν obtained in the third step with the text feature t obtained in the second step; for each head There are three weight matrices Used to calculate Query, Key and Value, satisfying: The attention weight is calculated using the scaled dot product attention mechanism, and the formula is as follows:
[0034]
[0035] Among them, d k Representing the dimension of each head, the calculated attention output is expressed as:
[0036]
[0037] The outputs of all heads are concatenated together and then passed through a linear layer, calculated as follows:
[0038] Z=Concat(Z1,Z2,…,Z h )W O
[0039] Among them, W O is the weight matrix of the linear layer; a feedforward network is then used to further process and transform the features output by the attention layer, which includes two fully connected layers, an activation layer, and a Dropout layer; the first fully connected layer transforms the feature dimension from D to 2×D, and the second fully connected layer restores the feature dimension from 2×D to D. The activation layer in the middle uses the ReLU activation function, which is calculated as follows:
[0040] FFN(Z)=W2max(0,W1Z+b1)+b2
[0041] Among them, w1, b1, W2, b2 are the parameters of the two fully connected layers; the final text vector t is obtained by using the residual connection between the multi-head attention output and the output of the feedforward network * , calculated as follows:
[0042] t * =Z+FFN(Z)
[0043] Use dynamically adaptive text vector t *Align with the corresponding image feature ν. This process uses hinge loss L2 for parameter optimization. The formula is as follows:
[0044]
[0045] Among them, ε represents the boundary parameter; using the pairwise similarity loss L3 to achieve the similar distribution of the Hamming distance and cosine distance in the hash space, the formula is as follows:
[0046]
[0047] Furthermore, in the fifth step, a balanced loss is designed to further constrain the optimization of hash network model parameters after the quantization loss L1, hinge loss L2 and pairwise similarity loss L3. The balanced loss L4 is calculated as follows:
[0048]
[0049] Among them, the parameter β is the balancing factor, and C is the new label set T generated after the weakly supervised label optimization module. * The number of categories in , Representative image x i In T * The one-hot encoding in t c Indicates T * The Cth text input is used to final The text vector generated by the weighted CLIP text encoder, γ is used as a regulation factor to adjust the degree of attention to more difficult samples; the balanced loss L4 dynamically adjusts the classification weights to achieve better learning of label samples with lower similarity.
[0050] Furthermore, in the sixth step, the hash codes of the query image and the database image are obtained through the trained network model, the Hamming distance between the hash codes is calculated and sorted, and finally the image set with the smallest distance is selected as the hash image retrieval result; where, it is assumed that the hash code of the query image generated by the model is h q , the hash code of the database image is h d ; The formula for calculating the Hamming distance between the two is as follows:
[0051]
[0052] Where b represents the number of bits of the hash code, h q [k] and h d [k] are hash codes h q and h d ; sort all database images according to the calculated Hamming distance, and select the first N images with the smallest distance as the final weakly supervised hash image retrieval results.
[0053] A weakly supervised hash image retrieval system based on a large-scale visual language model, including:
[0054] The fine-tuning module uses the training set containing weakly supervised labels to fine-tune the language image pre-training model CLIP for contrastive learning, and fuses the original pre-training weights and the fine-tuned weights through linear interpolation.
[0055] Weakly supervised label optimization module, used to replace noisy labels with labels that are more relevant to the image content;
[0056] The feature extraction module uses a pre-trained convolutional neural network to extract image features and maps the image features into feature vectors with the same dimensions as the hash code and text features;
[0057] The feature interaction module uses a multi-head self-attention mechanism to interact image features and text features and dynamically adjust the allocation of attention;
[0058] Loss function module, used to further constrain parameter updates in the hash network;
[0059] The hash image retrieval module obtains the hash codes of the query image and the database image through the trained hash network model, calculates the Hamming distance between the hash codes and sorts them, and finally selects the image set with the smallest distance as the hash image retrieval result.
[0060] Compared with the prior art, the present invention has the following significant features: (1) a weakly supervised label optimization module is designed to optimize noisy labels by utilizing the good cross-modal alignment capability of the large-scale visual language model CLIP, and to solve the problem that the fine-tuning process is affected by the long-tail distribution of downstream data and the mutual influence between classes in the inference stage; (2) a new balancing function is designed so that the model can pay more attention to the learning of difficult samples at the tail during training. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] Figure 1 It is a flow chart of a weakly supervised hash image retrieval method based on a large-scale visual language model of the present invention.
[0062] Figure 2 It is a network structure diagram of the present invention.
[0063] Figure 3 It is a schematic diagram of the result after the weak supervision label optimization in the present invention.
[0064] Figure 4 It is a structural diagram of a weakly supervised hash image retrieval system based on a large-scale visual language model of the present invention. DETAILED DESCRIPTION
[0065] The present invention uses the data of the training set to fine-tune the CLIP model and realizes the fusion of weights through linear interpolation. A weakly supervised label optimization module is designed to infer the labels before using multi-head self-attention to obtain a new label set with higher relevance to the image content. Finally, a balanced loss is designed so that the model can better focus on the learning of the more difficult samples at the tail during training, achieving high-precision hash image retrieval.
[0066] The present invention will be further described below in conjunction with the accompanying drawings.
[0067] Combination Figure 1 , the steps of weakly supervised hash image retrieval based on a large-scale visual language model of the present invention are described in detail.
[0068] The first step uses the training data S train ={(x i ,T i )|i=1,...,N tr} Fine-tune the CLIP model, where x i ∈R H ×W×3 represents the images in the training set, T i Represents the image x i The corresponding weakly supervised label set, N tr Represents the total number of pictures in the training set; the network structure of the present invention is as follows Figure 2 As shown, all the pictures in the training set are used as the visual input of the fine-tuning process. i Use the prompt template a photo of T i As its corresponding text description, where T i Each label in the label set is connected with spaces to form a string; in the fine-tuning process, cross entropy loss is used to optimize the parameters of both the image side and the text side, and the weight after fine-tuning is recorded as θ finetune , the original weight is recorded as θ zeroshot ; Linearly interpolate the original model weights with the fine-tuned model weights to obtain a new model weight denoted as θ final :
[0069] θ final =(1-α)·θ zeroshot +α·θ finetune
[0070] The interpolation coefficient α can control the contribution ratio of the model weight and is usually set to 0.5. Linear interpolation between weights can reduce the impact of uneven distribution of downstream data on model parameters during fine-tuning.
[0071] The second step is to design a weakly supervised label optimization module to optimize the noisy labels and reduce the mutual influence of redundant features between categories;
[0072] 2.1 In the process of optimizing the original weakly supervised labels, we first use the model weights θ normalized by the Euclidean norm final The obtained image features and text features are multiplied element by element to obtain the original cross-modal cross representation F o The calculation is as follows:
[0073]
[0074] where F i Represents the weight of the model used final Image features generated by the post-CLIP image encoder, F t Represents the weight of the model used final The text features generated by the post-CLIP text encoder; then the category labeling feature F is used c and text features F t The similarity score s is calculated as follows:
[0075]
[0076] The ratio of the similarity score of each category to the mean score is used as the weight w and is calculated as follows:
[0077]
[0078] Then, each feature is weighted by category, and the average is calculated in the category dimension as the redundant feature F. r , which can be described by the following formula:
[0079] F r =mean(F o ☉ expand(w)
[0080] By transforming the original cross-modal representation F o Remove redundant features F r A new cross-modal cross-representation F can be obtained n The calculation is as follows:
[0081] F n =F o -expand(F r )
[0082] Eliminating redundant features can reduce the mutual influence between categories when the model is predicting, thereby obtaining a more accurate category ranking. i The corresponding weakly supervised label set T iThe number of common labels in is α, then the top labels with the highest correlation with the image content in the inference stage are taken α The most optimized label is calculated as follows:
[0083]
[0084] 2.2 Optimize the training set S according to the process in 2.1 train Weakly supervised labels for all images in , such as Figure 3 As shown, we get the optimized label set T that filters out noise labels and reduces the influence of redundant features. * , using the model weights θ final The CLIP text encoder will then * The labels in are transformed into 768-dimensional text feature vectors t.
[0085] In the third step, the pre-trained AlexNet network is used to extract 4096-dimensional image features. In the hash network, the dimensions of the image features are mapped to the dimensions corresponding to the number of hash code bits through a fully connected layer, and the Sigmoid function is used for activation. This process uses the quantization loss L1 for parameter optimization, and the formula is as follows:
[0086]
[0087] Where m is the size of the minimum batch, b is the number of bits in the hash code, and h is i represents the hash code, Ι is the all-1 vector of the same dimension as the hash code; the obtained hash feature vector h is then passed through a fully connected layer again and activated using the LeakyReLU function to map the h feature dimension to a vector ν of the same dimension as t to achieve subsequent cross-modal feature interaction.
[0088] The fourth step uses the multi-head self-attention mechanism to interact the image feature ν obtained in the third step with the text feature t obtained in the second step; for each head There are three weight matrices Used to calculate Query, Key and Value, satisfying: The attention weight is calculated using the scaled dot product attention mechanism, and the formula is as follows:
[0089]
[0090] Among them, d k Representing the dimension of each head, the calculated attention output is expressed as:
[0091]
[0092] The outputs of all heads are concatenated together and then passed through a linear layer, calculated as follows:
[0093] Z=Concat(Z1,Z2,…,Z h )W O
[0094] Among them, W O is the weight matrix of the linear layer; a feedforward network is then used to further process and transform the features output by the attention layer, which includes two fully connected layers, an activation layer, and a Dropout layer; the first fully connected layer transforms the feature dimension from D to 2×D, and the second fully connected layer restores the feature dimension from 2×D to D. The activation layer in the middle uses the ReLU activation function, which is calculated as follows:
[0095] FFN(Z)=W2max(0,W1Z+b1)+b2
[0096] Among them, W1, b1, W2, b22 are the parameters of the two fully connected layers; the final text vector t is obtained by using the residual connection between the multi-head attention output and the output of the feedforward network * , calculated as follows:
[0097] t * =Z+FFN(Z)
[0098] Use dynamically adaptive text vector t * Align with the corresponding image feature ν. This process uses hinge loss L2 for parameter optimization. The formula is as follows:
[0099]
[0100] Among them, ε is set to 0.7 as a boundary parameter; the pairwise similarity loss L3 is used to achieve that the Hamming distance and cosine distance of the hash space have similar distributions, and the formula is as follows:
[0101]
[0102] In the fifth step, a balanced loss is designed to further constrain the model after the losses L1, L2, and L3, so that when there is a long-tail distribution in the training set, the model pays more attention to more difficult samples. The balanced loss L4 is calculated as follows:
[0103]
[0104] Among them, parameter β is the balancing factor, C is the new label set T generated after the weakly supervised label optimization module * The number of categories in , Representative image x i In T * The one-hot encoding in t c Indicates T *The Cth text input is used to final The text vector generated by the weighted CLIP text encoder, γ is used as a regulation factor to adjust the degree of attention to more difficult samples; the balanced loss L4 dynamically adjusts the classification weights to achieve better learning of label samples with lower similarity;
[0105] The sixth step is to obtain the hash codes of the query image and the database image through the trained network model, calculate the Hamming distance between the hash codes and sort them, and finally select the image set with the smallest distance as the hash image retrieval result; where, assuming that the hash code of the query image generated by the model is h q , the hash code of the database image is h d ; Then, the formula for calculating the Hamming distance between the two is as follows:
[0106]
[0107] Where b represents the number of bits of the hash code, h q [k] and h d [k] are hash codes h q and h d ; sort all database images according to the calculated Hamming distance, and select the first N images with the smallest distance as the final weakly supervised hash image retrieval results.
[0108] Combination Figure 4 The present invention provides a weakly supervised hash image retrieval system based on a large-scale visual language model, comprising:
[0109] The fine-tuning module uses the training set containing weakly supervised labels to fine-tune the language image pre-training model CLIP for contrastive learning, and fuses the original pre-training weights and the fine-tuned weights through linear interpolation.
[0110] Weakly supervised label optimization module, used to replace noisy labels with labels that are more relevant to the image content;
[0111] The feature extraction module uses a pre-trained convolutional neural network to extract image features and maps the image features into feature vectors with the same dimensions as the hash code and text features;
[0112] The feature interaction module uses a multi-head self-attention mechanism to interact image features and text features and dynamically adjust the allocation of attention;
[0113] Loss function module, used to further constrain parameter updates in the hash network;
[0114] The hash image retrieval module obtains the hash codes of the query image and the database image through the trained hash network model, calculates the Hamming distance between the hash codes and sorts them, and finally selects the image set with the smallest distance as the hash image retrieval result.
[0115] Experimental Results
[0116] The present invention verifies the performance of the model for hash image retrieval through the mAP value. The larger the mAP value, the better the performance of the model. The proposed weakly supervised hash image retrieval method based on a large-scale visual language model is compared with the currently advanced international multi-label classification methods, including WDHT, MGRN, EWSH and WSHRCA methods.
[0117] Tables 1 and 2 show the mAP values calculated by the present invention and the above four models in the MirFlicker and NUS-WIDE datasets using all the returned results. The MirFlicker dataset contains 38 semantic labels for testing and 1,386 common weakly supervised labels provided by users, from which 15,240 images are randomly selected as training sets and datasets, and 2,000 images are used as query sets. The NUS-WIDE dataset uses the 10 most common categories for testing and 638 common weakly supervised labels for training, from which 15,240 images are randomly selected as training sets, 2,000 images are used as query sets, and the remaining images are used as datasets.
[0118] Table 1 Comparison of MirFlicker experimental results
[0119]
[0120] Note: Bold fonts indicate the best result in each row
[0121] Table 2 Comparison of NUS-WIDE experimental results
[0122]
[0123] Note: Bold fonts indicate the best result in each row
[0124] From the comparison results in Tables 1 and 2, it can be seen that the present invention improves the retrieval performance under different bit numbers in the MirFlicker and NUS-WIDE datasets.
Claims
1. A weakly supervised hash image retrieval method based on a large-scale visual language model, It is characterized in that The following steps are involved: Step 1: Fine-tune the CLIP language image pre-training model for contrastive learning using a training set containing weakly supervised labels, and fuse the original pre-trained weights and the fine-tuned weights through linear interpolation. Step 2: Design a weakly supervised label optimization module to replace noisy labels with labels that are more relevant to the image content; Step 3: Use a pre-trained convolutional neural network to extract image features, and then train a hash network to map image features into feature vectors of the same dimensions as the hash code and text features; Step 4: Use the multi-head self-attention mechanism to interact image features and text features and dynamically adjust the allocation of attention; Step 5: Design a balanced loss to further constrain parameter updates in the hash network; Step 6: Obtain the hash codes of the query image and database image through the trained hash network model, calculate the Hamming distance between the hash codes and sort them, and finally select the image set with the smallest distance as the hash image retrieval result.
2. The weakly supervised hash image retrieval method based on a large-scale visual language model according to claim 1, Features: The first step uses the training data S train ={(x i ,T i )|i=1,…,N tr } Fine-tune the CLIP model, where x i ∈R H×W×3 represents the images in the training set, T i Represents image x i The corresponding weakly supervised label set, N tr Represents the total number of images in the training set; use all images in the training set as the visual input of the fine-tuning process. For image x i Use the prompt template a photo of T i As its corresponding text description, where T i Each label in the label set is connected into a string with spaces; in the fine-tuning process, cross entropy loss is used to optimize the parameters of both the image side and the text side, and the weight after fine-tuning is recorded as θ finetune , the original weight is recorded as θ zeroshot ; Linearly interpolate the original model weights with the fine-tuned model weights to obtain a new model weight denoted as θ final : i final =(1-α)·θ zeroshot +a·θ finetune Among them, the interpolation coefficient α is used to control the contribution ratio of the model weight.
3. The weakly supervised hash image retrieval method based on a large-scale visual language model according to claim 1, Features: The second step is to design a weakly supervised label optimization module to optimize the noisy labels; 2.1 In the process of optimizing the original weakly supervised labels, we first use the model weights θ normalized by the Euclidean norm final The obtained image features and text features are multiplied element by element to obtain the original cross-modal cross representation F o , calculated as follows: where F i Represents the weight of the model used final Image features generated by the post-CLIP image encoder, F t Represents the weight of the model used final Text features generated by the post-CLIP text encoder; Then use the class label feature F c and text features F t The similarity score s is calculated as follows: The ratio of the similarity score of each category to the mean score is used as the weight w and is calculated as follows: Then, each feature is weighted by category, and the average is calculated in the category dimension as the redundant feature F. r , the formula is described as follows: F r =mean(F o ⊙expand(w)) By transforming the original cross-modal representation F o Remove redundant features F r , and obtain a new cross-modal cross representation F n The calculation is as follows: F n =F o -expand(F r ) Assume that image x i The corresponding weakly supervised label set T o The number of common labels in is α, then take the top labels with the highest correlation with the image content. α The optimized labels are calculated as follows: 2.2 Optimize the training set S according to the process in 2.1 train Weakly supervised labels of all images in the image are obtained to obtain the optimized label set T that filters out noise labels and reduces the influence of redundant features. * , using the model weights θ final The CLIP text encoder will then * The labels in are transformed into text feature vectors t.
4. The weakly supervised hash image retrieval method based on a large-scale visual language model according to claim 1, It is characterized in that The third step is to use the pre-trained convolutional neural network to extract image features; in the hash network, the dimension of the image features is mapped to the dimension corresponding to the number of hash code bits through a fully connected layer, and the Sigmoid function is used for activation; this process uses the quantization loss L 1 Parameter optimization is performed, and the formula is as follows: Where m is the size of the minimum batch, b is the number of bits in the hash code, and h is i represents the hash code, Ι is the all-1 vector of the same dimension as the hash code; the obtained hash feature vector j is then passed through a fully connected layer again and activated using the leakyReLU function to map the h feature dimension to a vector ν of the same dimension as t to achieve subsequent cross-modal feature interaction.
5. The weakly supervised hash image retrieval method based on a large-scale visual language model according to claim 1, Features: The fourth step uses the multi-head self-attention mechanism to interact the image feature ν obtained in the third step with the text feature t obtained in the second step; for each head There are three weight matrices Used to calculate Query, Key and Value, satisfying: The attention weight is calculated using the scaled dot product attention mechanism, and the formula is as follows: Among them, d k Representing the dimension of each head, the calculated attention output is expressed as: The outputs of all heads are concatenated together and then passed through a linear layer, calculated as follows: Z=Concat(Z 1 ,WITH 2 ,…,WITH h )IN O Among them, W O is the weight matrix of the linear layer; a feedforward network is then used to further process and transform the features output by the attention layer, which includes two fully connected layers, an activation layer, and a Dropout layer; the first fully connected layer transforms the feature dimension from D to 2×D, and the second fully connected layer restores the feature dimension from 2×D to D. The activation layer in the middle uses the ReLU activation function, which is calculated as follows: FFN(Z)=W 2 max(0,W 1 Z+b 1 )+b 2 Among them, W 1 ,b 1 ,W 2 ,b 2 are the parameters of the two fully connected layers; the final text vector t is obtained by using the residual connection between the multi-head attention output and the output of the feedforward network * , calculated as follows: t * =Z+FFN(Z) Use dynamically adaptive text vector t * Align with the corresponding image feature ν, this process uses the hinge loss L 2 Parameter optimization is performed, and the formula is as follows: Among them, ε represents the boundary parameter; the pairwise similarity loss L is used 3 The Hamming distance of the hash space has a similar distribution to the cosine distance, and the formula is as follows:
6. The weakly supervised hash image retrieval method based on a large-scale visual language model according to claim 1, Features: The fifth step is to design a balanced loss, in the quantization loss L 1 , hinge loss L 2 and pairwise similarity loss L 3 Then further constrain the optimization of hash network model parameters to balance the loss L 4 The calculation is as follows: Among them, the parameter β is the balancing factor, and C is the new label set T generated after the weakly supervised label optimization module. * The number of categories in , Representative image x i In T * The one-hot encoding in t c Indicates T * The Cth text input is used to final The text vector generated by the weighted CLIP text encoder, γ is used as a regulation factor to adjust the degree of attention to more difficult samples; the balance loss L 4 By dynamically adjusting the classification weights, it is possible to better learn label samples with lower similarity.
7. The weakly supervised hash image retrieval method based on a large-scale visual language model according to claim 1, It is characterized in that The sixth step is to obtain the hash codes of the query image and the database image through the trained network model, calculate the Hamming distance between the hash codes and sort them, and finally select the image set with the smallest distance as the hash image retrieval result; where, assuming that the hash code of the query image generated by the model is h q , the hash code of the database image is h d ; The formula for calculating the Hamming distance between the two is as follows: Where b represents the number of bits of the hash code, h q [k] and h d [k] are hash codes h q and h d ; sort all database images according to the calculated Hamming distance, and select the first N images with the smallest distance as the final weakly supervised hash image retrieval results.
8. A weakly supervised hash image retrieval system based on a large-scale visual language model, Features: include The fine-tuning module uses the training set containing weakly supervised labels to fine-tune the language image pre-training model CLIP for contrastive learning, and fuses the original pre-training weights and the fine-tuned weights through linear interpolation. Weakly supervised label optimization module, used to replace noisy labels with labels that are more relevant to the image content; The feature extraction module uses a pre-trained convolutional neural network to extract image features and maps the image features into feature vectors with the same dimensions as the hash code and text features; The feature interaction module uses a multi-head self-attention mechanism to interact image features and text features and dynamically adjust the allocation of attention; Loss function module, used to further constrain parameter updates in the hash network; The hash image retrieval module obtains the hash codes of the query image and the database image through the trained hash network model, calculates the Hamming distance between the hash codes and sorts them, and finally selects the image set with the smallest distance as the hash image retrieval result.
Citation Information
Patent Citations
Deep unsupervised image retrieval method based on sparse graph structure
CN110941734A
Zero-sample image retrieval method based on attribute-guided adversarial hash network
CN112364195A
Depth cross-modal hash image retrieval method based on joint semantic matrix
CN113177132A
Cross-modal data processing method and device, storage medium and electronic device
WO2022068195A1
Cross-modal hash retrieval method, terminal device, and storage medium
WO2022104540A1
Cited By
Large language model fingerprint adding method and device based on weight superposition
CN119598433A
Investor emotion index verification method based on machine learning
CN119988592A
Self-adaptive medical visual question-answering method and device and medium
CN120973965A
