Cross-modal retrieval method and system based on robust double-hash consensus learning

Abstract

The invention provides a cross-modal retrieval method and system based on robust double-hash consensus learning, and relates to the field of data processing.The method comprises the steps that an image modal network, a text modal network and a prototype are constructed; extracting image features of the multi-modal sample pair through an image modal network; extracting text features of the multi-modal sample pair through a text modal network; through the prototype, obtaining a prediction soft label of the multi-modal sample pair; determining a prediction confidence coefficient based on the prediction soft label and the original label of the multi-modal sample pair, and performing label correction on the prediction soft label; updating the parameters of the image modal network, the text modal network and the prototype by using the image hash code and the text hash code of the multi-modal sample pair and the corrected label, and generating a cross-modal hash retrieval model; through the cross-modal Hash retrieval model, the cross-modal retrieval result corresponding to the cross-modal retrieval request is obtained, and the method has the advantage of improving the accuracy and robustness of cross-modal retrieval.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a cross-modal retrieval method and system based on robust double hash consensus learning. Background Art

[0002] In recent years, with the rapid development of big data and artificial intelligence, cross-modal hashing has become a research hotspot for multimodal data retrieval. Cross-modal retrieval involves mapping different modalities (such as images, text, and video) into a unified representation space, enabling cross-retrieval and matching of multimodal data. Traditional cross-modal retrieval typically relies on the computation and storage of high-dimensional features and often requires the storage of large feature vectors, resulting in significant computational and storage overhead and low retrieval efficiency. Different modalities (such as images and text) have distinct feature spaces, and traditional methods often rely on complex mapping functions, which often yields poor results when the inter-modal disparity is significant. To address these issues, cross-modal hashing has emerged. By mapping high-dimensional data into the same low-dimensional hash code space, cross-modal hashing not only improves inter-modal alignment but also significantly reduces computational complexity and storage requirements, thereby enhancing retrieval efficiency. Initial research primarily relied on traditional machine learning methods, manually designing features and constructing inter-modal similarity mappings using metric learning or deep neural networks. However, due to the significant differences in feature distributions across modal data, these methods are limited in retrieval efficiency and accuracy in large-scale data scenarios. With the rise of deep learning, hash learning methods based on neural networks have gradually become mainstream. By constructing a deep feature extraction network and combining it with hash coding technology, different modal data are mapped to a shared low-dimensional hash space, thereby achieving efficient retrieval.

[0003] Expanding from single-modality hashing to cross-modal hashing, and exploring more effective feature alignment strategies, such as optimization methods based on adversarial learning, attention mechanisms, and contrastive learning, has improved representation consistency and retrieval accuracy across modalities. However, in cross-modal retrieval tasks, noisy labels in the dataset are incorrectly or inconsistently labeled, often due to human error or automated labeling tools. These noisy labels can interfere with model training, causing the model to learn incorrect information, leading to noise overfitting and degrading performance.

[0004] Therefore, it is necessary to provide a cross-modal retrieval method and system based on robust double hash consensus learning, which can be used to improve the accuracy and robustness of cross-modal retrieval through label correction. Summary of the Invention

[0005] The present invention provides a cross-modal retrieval method based on robust double hash consensus learning, comprising: constructing an image modal network, a text modal network and a prototype; extracting image features of image modal samples in a multimodal sample pair through the image modal network; extracting text features of text modal samples in the multimodal sample pair through the text modal network; obtaining predicted soft labels of the multimodal sample pair through the prototype; determining the prediction confidence of the multimodal sample pair based on the predicted soft labels and original labels of the multimodal sample pair; performing label correction on the predicted soft labels of the multimodal sample pair based on the prediction confidence of the multimodal sample pair; updating parameters of the image modal network, the text modal network and the prototype using the image hash code, the text hash code and the corrected labels of the multimodal sample pair; generating a cross-modal hash retrieval model based on the parameters of the image modal network, the text modal network and the prototype after multiple rounds of training; and obtaining a cross-modal retrieval result corresponding to the cross-modal retrieval request through the cross-modal hash retrieval model.

[0006] Furthermore, the image modality network includes an image modality backbone network, an image modality dual-branch structure and an image modality output layer, the image modality dual-branch structure includes an image modality first branch and an image modality second branch, and the image modality first branch and the image modality second branch share weights; the image modality backbone network is used to extract initial image features from the input image; the image modality first branch includes a multi-layer image modality first fully connected layer, and the image modality first branch is used to convert the initial image features into an image modality first embedding vector; the image modality second branch includes a multi-layer image modality second fully connected layer, and the image modality first branch is used to convert the initial image features into an image modality second embedding vector; the image modality output layer is used to normalize the image modality first embedding vector and the image modality second embedding vector to generate image features of the image modality samples in the multimodal sample pair, wherein the image features include a first image feature vector corresponding to the image modality first embedding vector and a second image feature vector corresponding to the image modality second embedding vector.

[0007] Furthermore, the text modality network includes a text modality backbone network, a text modality dual-branch structure and a text modality output layer, the text modality dual-branch structure includes a text modality first branch and a text modality second branch, and the text modality first branch and the text modality second branch share weights; the text modality backbone network is used to convert the input text data into a dense vector representation; the text modality first branch includes a multi-layer text modality first fully connected layer, and the text modality first branch is used to convert the initial image features into a text modality first embedding vector; the text modality second branch includes a multi-layer image modality second fully connected layer, and the text modality first branch is used to convert the initial image features into a text modality second embedding vector; the text modality output layer is used to normalize the text modality first embedding vector and the text modality second embedding vector to generate text features of the text modality samples in the multimodal sample pair, wherein the text features include a first text feature vector corresponding to the text modality first embedding vector and a second text feature vector corresponding to the text modality second embedding vector.

[0008] Furthermore, the method of obtaining predicted soft labels for multimodal sample pairs through the prototype includes: generating a first feature vector group and a second feature vector group based on the first image feature vector, the second image feature vector, the first text feature vector and the second text feature vector, wherein the first feature vector group includes one of the first image feature vector and the second image feature vector, the second feature vector group includes the other of the first image feature vector and the second image feature vector, the first feature vector group also includes one of the first text feature vector and the second text feature vector, and the second feature vector group also includes the other of the first text feature vector and the second text feature vector; obtaining a first predicted soft label corresponding to the first feature vector group through the prototype; obtaining a second predicted soft label corresponding to the second feature vector group through the prototype; and obtaining predicted soft labels for the multimodal sample pairs based on the first predicted soft label and the second predicted soft label.

[0009] Furthermore, the first predicted soft label corresponding to the first feature vector group is obtained through the prototype, including: S11, obtaining a predicted value based on the first feature vector group through the prototype; S12, obtaining an image probability distribution and a text probability distribution based on the predicted value; S13, initializing the current number of iterations T=1; S14, obtaining an intermediate predicted soft label for the T+1 iteration based on the intermediate predicted soft label, image probability distribution and text probability distribution of the T-th iteration, when T=1, the intermediate predicted soft label for the T-th iteration is the original label; S15, judging whether the iteration end condition is met, if so, completing the iteration, and taking the intermediate predicted soft label for the T-th iteration as the first predicted soft label corresponding to the first feature vector group, if not, setting T=T+1 and executing S14.

[0010] Furthermore, the predicted soft labels of the multimodal sample pairs are subjected to label correction based on the prediction confidence of the multimodal sample pairs, including: determining the target category index of the multimodal sample pairs based on the prediction probability of each category index in the first predicted soft label and the prediction probability of each category index in the second predicted soft label; generating the final predicted label of the multimodal sample pairs based on the target category index; and performing label correction on the predicted soft labels of the multimodal sample pairs based on the prediction confidence, the final predicted label and the original label of the multimodal sample pairs.

[0011] Furthermore, the image hash codes, text hash codes and corrected labels of the multimodal sample pairs are used to update the parameters of the image modality network, the text modality network and the prototype, including: using a cross-modal joint loss function and a prototype loss function to update the parameters of the image modality network, the text modality network and the prototype based on the image hash codes, text hash codes and corrected labels of the multimodal sample pairs.

[0012] Furthermore, the cross-modal joint loss function is:

[0013]

[0014] in, is the cross-modal joint loss function, is the similarity comparison loss function, λ is the weight, Q is the regularization term, m is the threshold, n is the number of sample logarithms, θ is the temperature coefficient, and Y ij is the label matrix of the image sample in the i-th sample pair and the text sample in the j-th sample pair, is the similarity between the image sample in the i-th sample pair and the text sample in the j-th sample pair, is the similarity between the text sample in the i-th sample pair and the image sample in the j-th sample pair, i,j∈{1,2,…,n}.

[0015] Furthermore, the prototype loss function is:

[0016]

[0017]

[0018] in, is the prototype loss function, is the prototype alignment loss function, is the mean square error loss function, Y' ij is the label matrix after linear change of the i-th sample pair and the j-th category, is the similarity between the image sample in the i-th sample pair and the prototype of the j-th category, is the similarity between the text sample in the i-th sample pair and the prototype of the j-th category, R ij is the similarity between the prototype of the i-th category and the prototype of the j-th category, A ij is the centralized matrix, A ij The diagonal position is 1, the rest of the positions are -1, and the size is C*C, where C is the number of categories.

[0019] The present invention provides a cross-modal retrieval system based on robust double hash consensus learning, which applies the above-mentioned cross-modal retrieval method based on robust double hash consensus learning, including: a model training module for constructing an image modality network, a text modality network and a prototype; extracting image features of image features of image modality samples in a multimodal sample pair through the image modality network; extracting text features of text features of text modality samples in a multimodal sample pair through the text modality network; obtaining predicted soft labels of the multimodal sample pair through the prototype; and predicting soft labels based on the multimodal sample pair. and original labels to determine the prediction confidence of the multimodal sample pair; based on the prediction confidence of the multimodal sample pair, the predicted soft labels of the multimodal sample pair are corrected; the image hash code, text hash code and corrected labels of the multimodal sample pair are used to update the parameters of the image modality network, text modality network and prototype; according to the parameters of the image modality network, text modality network and prototype after multiple rounds of training, a cross-modal hash retrieval model is generated; the cross-modal retrieval module is used to obtain the cross-modal retrieval results corresponding to the cross-modal retrieval request through the cross-modal hash retrieval model.

[0020] Compared with the existing technology, the cross-modal retrieval method and system based on robust double hash consensus learning provided by the present invention have at least the following beneficial effects:

[0021] 1. By utilizing cross-modal hashing, the problem of excessive high-dimensional feature computation and storage overhead in traditional cross-modal retrieval methods is overcome. The double-hash consensus learning mechanism solves the problem of single feature expression, improves the diversity of feature learning, enhances the modal alignment effect, and combines label correction strategies to reduce noise interference, thereby improving the accuracy and robustness of cross-modal retrieval.

[0022] 2. Cross-modal hashing is used to map data from different modalities into the same low-dimensional hash space, so that the features between modalities are more closely aligned, improving the matching accuracy of cross-modal data. A dual hashing network structure is used to extract modal features from different angles, avoiding the limitations of single model feature learning, and making the model perform better in complex data scenarios.

[0023] 3. By correcting incorrect or inconsistent labels and reducing noise interference, the model can learn more realistic cross-modal features, improve generalization ability and retrieval accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] This specification will be further described in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, like numbers represent like structures, wherein:

[0025] Figure 1 is a flowchart of a cross-modal retrieval method based on robust double hash consensus learning according to some embodiments of this specification;

[0026] Figure 2 is a schematic diagram of the structure of an image modality network according to some embodiments of this specification;

[0027] Figure 3 is a schematic diagram of generating a confidence mask according to the double hash consensus technology shown in some embodiments of this specification;

[0028] Figure 4 is a schematic diagram of generating weighted labels according to the double hash consensus technology shown in some embodiments of this specification;

[0029] Figure 5 This is a module diagram of a cross-modal retrieval system based on robust double hash consensus learning according to some embodiments of this specification. DETAILED DESCRIPTION

[0030] To more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly describes the drawings required for describing the embodiments. Obviously, the drawings described below are merely examples or embodiments of this specification. Those skilled in the art can apply this specification to other similar scenarios based on these drawings without inventive effort. Unless otherwise apparent from the context or otherwise noted, the same reference numerals in the figures represent the same structure or operation.

[0031] Figure 1 is a flow chart of a cross-modal retrieval method based on robust double hash consensus learning according to some embodiments of this specification, such as Figure 1 As shown, the cross-modal retrieval method based on robust double hashing consensus learning can include the following steps.

[0032] Step 110: construct an image modality network, a text modality network, and a prototype.

[0033] Figure 2 is a schematic diagram of the structure of an image modality network according to some embodiments of this specification, such as Figure 2As shown, in some embodiments, the image modality network includes an image modality backbone network, an image modality dual-branch structure, and an image modality output layer, the image modality dual-branch structure includes an image modality first branch and an image modality second branch, and the image modality first branch and the image modality second branch share weights;

[0034] The image modality backbone network is used to extract initial image features from the input image. Specifically, the image modality backbone network may include a pre-trained VGG (Visual Geometry Group, VGG)19 model;

[0035] The image modality first branch includes a multi-layer image modality first fully connected layer, and the image modality first branch is used to convert the initial image features into an image modality first embedding vector;

[0036] The image modality second branch includes a multi-layer image modality second fully connected layer, and the image modality first branch is used to convert the initial image features into an image modality second embedding vector;

[0037] The image modality output layer is used to normalize the image modality first embedding vector and the image modality second embedding vector to generate image features of the image modality samples in the multimodal sample pair, ensuring output at the same scale, wherein the image features include a first image feature vector corresponding to the image modality first embedding vector and a second image feature vector corresponding to the image modality second embedding vector. If it is a training mode, the first image feature vector corresponding to the image modality first embedding vector and the second image feature vector corresponding to the image modality second embedding vector are output for comparison and calculation of the loss function; if it is not a training mode, the average value of the first image feature vector corresponding to the image modality first embedding vector and the second image feature vector corresponding to the image modality second embedding vector is output.

[0038] Specifically, the structure of the first branch of the image modality is consistent with the structure of the second branch of the image modality. The following explanation is given by taking the first branch of the image modality as an example.

[0039] The first fully connected layer of the multi-layer image modality can include at least a linear transformation layer, a ReLU (Rectified Linear Unit) activation layer and a Dropout layer, which is used to further map the initial image features extracted by the image modality backbone network to an embedding space of a specified dimension and output an embedding representation of the same dimension as the input image.

[0040] In some embodiments, the text modality network includes a text modality backbone network, a text modality dual-branch structure, and a text modality output layer, the text modality dual-branch structure includes a text modality first branch and a text modality second branch, and the text modality first branch and the text modality second branch share a weight;

[0041] The text modality backbone network is used to convert the input text data into a dense vector representation. Specifically, the text modality backbone network may include a pre-trained Doc2Vec model;

[0042] The text modality first branch includes multiple layers of text modality first fully connected layers, and the text modality first branch is used to convert the initial image features into a text modality first embedding vector;

[0043] The text modality second branch includes a multi-layer image modality second fully connected layer, and the text modality first branch is used to convert the initial image features into the text modality second embedding vector;

[0044] The text modality output layer is used to normalize the text modality first embedding vector and the text modality second embedding vector to generate text features of the text modality samples in the multimodal sample pair, wherein the text features include a first text feature vector corresponding to the text modality first embedding vector and a second text feature vector corresponding to the text modality second embedding vector.

[0045] Specifically, the structures of the first branch of the text modality and the second branch of the text modality are similar to the structure of the first branch of the image modality, and are not described in detail here.

[0046] It can be understood that both the image modality network and the text modality network use a dual-branch neural network structure with shared weights. The input image and text data are processed by two identical network paths respectively, so that the feature representation of each modality is generated at the same scale. This allows the image modality network and the text modality network to extract features from different perspectives and to a certain extent complement redundant information, which helps the model learn more robust, consistent and generalizable feature representations.

[0047] The prototype, that is, the learnable parameter matrix W, is used to map the input image and text features to the hash space. The dimension of the learnable parameter matrix W is (n class ,bit len ), where n class is the number of categories, bit len is the length of the hash code; the prototype initialization method uses orthogonal initialization. In image-text joint hashing, the prototype needs to adapt to the distribution of both image and text features to ensure that images and texts with the same semantics are mapped to similar hash codes.

[0048] As you can understand, the prototype maps each category to a central point, providing a global representation of the category. This representation is based on the core features of the category, allowing the model to understand each category from a global perspective and push samples of different categories further apart, making the boundaries between categories clearer.

[0049] Prototype optimization optimizes modeling for each category, enabling features from different modalities to find corresponding, optimal representations within a shared space. Category prototypes for images and text are optimized to align within the same semantic space, enabling better matching and comparison of samples of the same category across modalities. By optimizing category representations, prototypes can reduce differences between modalities, thereby improving retrieval accuracy.

[0050] During the training process, prototype optimization continuously adjusts the central representation of each category, allowing the model to focus on the core features of the category rather than individual abnormal samples. This enables the generated cross-modal hash retrieval model to maintain stable category recognition in noisy environments, thereby improving robustness and performance on noisy datasets.

[0051] Step 120: extracting image features of the image modality samples in the multimodal sample pairs through the image modality network.

[0052] Step 130: extract text features of the text modality samples in the multimodal sample pairs through the text modality network.

[0053] Step 140: Obtain predicted soft labels for multimodal sample pairs through the prototype.

[0054] In some embodiments, step 140 specifically includes:

[0055] generating a first feature vector group and a second feature vector group based on the first image feature vector, the second image feature vector, the first text feature vector, and the second text feature vector, wherein the first feature vector group includes one of the first image feature vector and the second image feature vector, the second feature vector group includes the other of the first image feature vector and the second image feature vector, the first feature vector group also includes one of the first text feature vector and the second text feature vector, and the second feature vector group also includes the other of the first text feature vector and the second text feature vector;

[0056] Obtaining a first predicted soft label corresponding to the first feature vector group through the prototype;

[0057] Obtain a second predicted soft label corresponding to the second feature vector group through the prototype;

[0058] Based on the first predicted soft label and the second predicted soft label, a predicted soft label of the multimodal sample pair is obtained.

[0059] In some embodiments, obtaining a first predicted soft label corresponding to the first feature vector group through the prototype includes:

[0060] S11. Obtain a predicted value based on the first feature vector group through the prototype;

[0061] S12. Based on the predicted values, obtain image probability distribution and text probability distribution;

[0062] S13, initialize the current number of iterations T = 1;

[0063] S14. Based on the intermediate predicted soft label of the T-th iteration, the image probability distribution, and the text probability distribution, obtain the intermediate predicted soft label of the T+1-th iteration. When T=1, the intermediate predicted soft label of the T-th iteration is the original label.

[0064] S15. Determine whether the iteration end condition is met, for example, the number of iterations reaches the maximum value or the intermediate predicted soft label converges. If so, the iteration is completed, and the intermediate predicted soft label of the Tth iteration is used as the first predicted soft label corresponding to the first feature vector group. If not, set T = T + 1 and execute S14.

[0065] Specifically, the predicted value can be obtained according to the following formula:

[0066]

[0067] Where z is the predicted value, f is the first eigenvector group, τ is the temperature coefficient, and tanh() is the activation function.

[0068] The image probability distribution and text probability distribution can be calculated according to the following formula:

[0069] p u =softmax(z u )

[0070] p v =softmax(z v )

[0071] Among them, p u is the image probability distribution, z u is the image feature vector included in the first feature vector group, softmax() is the activation function, p v is the text probability distribution, z v is the text feature vector included in the first feature vector group.

[0072] The intermediate predicted soft label for the T+1th iteration can be obtained according to the following formula:

[0073]

[0074] in, is the intermediate predicted soft label of the T+1th iteration, y last is the intermediate predicted soft label of the Tth iteration, and α is a hyperparameter used to control the balance between the previous round of intermediate predicted soft labels and the current prediction.

[0075] The method for obtaining the second predicted soft label corresponding to the second feature vector group through the prototype is the same as the method for obtaining the first predicted soft label corresponding to the first feature vector group through the prototype, and will not be repeated here.

[0076] As you can understand, using prototypes to generate predicted soft labels corresponding to the first and second feature vector groups helps reduce overfitting and improve the model's generalization and stability. Furthermore, when using prototypes to predict soft labels in each round of training, the model calculates new soft labels based on a weighted average of the current image or text features and the soft labels predicted in the previous round. This process enables the model to gradually adjust and optimize the labels, making them smoother and more stable.

[0077] For the first predicted soft label, the class index with the highest predicted probability may be selected to determine a first predicted label vector corresponding to the first predicted soft label. For the second predicted soft label, the class index with the highest predicted probability may be selected to determine a second predicted label vector corresponding to the second predicted soft label. The predicted soft labels for the multimodal sample pair may include the first predicted label vector and the second predicted label vector.

[0078] For example, the first predicted label vector corresponding to the first predicted soft label can be determined according to the following formula:

[0079]

[0080] in, is the first predicted label vector corresponding to the first predicted soft label, To obtain the maximum value, is the predicted probability corresponding to the kth category corresponding to the first eigenvector group.

[0081] The method for determining the second predicted label vector corresponding to the second predicted soft label is similar to the method for determining the first predicted label vector corresponding to the first predicted soft label, and will not be repeated here.

[0082] Step 150 : Determine the prediction confidence of the multimodal sample pair based on the predicted soft label and the original label of the multimodal sample pair.

[0083] Figure 3 is a schematic diagram of generating a confidence mask according to the double hash consensus technology shown in some embodiments of this specification, such as Figure 3 As shown, specifically, if there is at least one predicted label vector in the predicted soft labels of the multimodal sample pair that matches the original label, then the prediction confidence of the multimodal sample pair is high; if both predicted label vectors do not match the original label, then the prediction confidence of the sample is low; the expression of confidence C is:

[0084]

[0085] It can be understood that by comparing the first predicted label vector and the second predicted label vector included in the predicted soft label of the multimodal sample pair with the original label, more information and prediction perspectives can be obtained, making the calculated confidence indicator more reliable.

[0086] Step 160 : performing label correction on the predicted soft labels of the multimodal sample pairs based on the prediction confidence of the multimodal sample pairs.

[0087] In some embodiments, step 160 specifically includes:

[0088] Determining a target category index for the multimodal sample pair based on the predicted probability of each category index in the first predicted soft label and the predicted probability of each category index in the second predicted soft label;

[0089] Generate the final predicted label of the multimodal sample pair based on the target category index;

[0090] Based on the prediction confidence, final prediction label and original label of the multimodal sample pair, the predicted soft label of the multimodal sample pair is corrected.

[0091] Specifically, the predicted probability of each category index in the first predicted soft label and the predicted probability of each category index in the second predicted soft label can be weighted and summed to determine the target category index of the multimodal sample pair. The target category index can be determined according to the following formula:

[0092]

[0093] in, is the target category index, is the predicted probability of the k-th category index in the first predicted soft label, is the predicted probability of the k-th category index in the second predicted soft label, Used to select the category index corresponding to the maximum probability after summation as the target category index.

[0094] The final predicted label of the multimodal sample pair can be generated according to the following formula:

[0095]

[0096] in, is the final predicted label of the multimodal sample pair, Is a unit vector, which is the vector form of the target category index.

[0097] Figure 4This is a schematic diagram of generating weighted labels based on the double hash consensus technology shown in some embodiments of this specification, such as Figure 4 As shown in the figure, weighting is performed based on the confidence mask: the label matrix is weighted. If the confidence of the multimodal sample pair is high, then the label of the multimodal sample pair will rely more on the predicted label; for the multimodal sample pair with low confidence, the original label is retained; the weighted label is the final label after label correction. For example, the predicted soft label of the multimodal sample pair can be corrected according to the following formula:

[0098]

[0099] in, is the corrected label, Y k is the original label, C k is the confidence level, which takes the value of 0 or 1.

[0100] It is understandable that by introducing the confidence mask, label correction can dynamically adjust the weights of the final predicted label and the original label according to the confidence of the multimodal sample pair; for high-confidence samples, it relies more on the model's prediction results; while for low-confidence samples, the original label is still retained. This weighting mechanism makes training more flexible and accurate. In addition, during the training process, the labels of some multimodal sample pairs may affect the learning effect of the model due to noise, modality inconsistency or ambiguity; through label correction, the impact of low-confidence samples can be effectively reduced, thereby avoiding overfitting or interference of incorrect labels on model training; in the case of inaccurate sample labeling or noise in the sample itself, by combining the original label with the predicted soft label, label correction can provide more accurate information for the model, which helps to improve the accuracy of the training process.

[0101] Step 160 , using the image hash code, text hash code and corrected label of the multimodal sample pair to update the parameters of the image modality network, the text modality network and the prototype.

[0102] In some embodiments, step 160 specifically includes:

[0103] Using the cross-modal joint loss function and the prototype loss function, the parameters of the image modality network, text modality network and prototype are updated based on the image hash code, text hash code and rectified label of the multimodal sample pairs.

[0104] Among them, the cross-modal joint loss function is:

[0105]

[0106] in, is the cross-modal joint loss function, is the similarity comparison loss function, λ is the weight, Q is the regularization term, m is the threshold, n is the number of sample logarithms, θ is the temperature coefficient, and Y ij is the label matrix of the image modality sample in the i-th multimodal sample pair and the text modality sample in the j-th multimodal sample pair, is the similarity between the image sample in the i-th sample pair and the text sample in the j-th sample pair, is the similarity between the text sample in the i-th sample pair and the image sample in the j-th sample pair, i,j∈{1,2,…,n}. and The calculation formula is:

[0107]

[0108] Among them, ξ is a hyperparameter that penalizes dissimilar sample pairs and reduces their contribution to the loss function; m is a threshold that controls the distance between the boundary and the sample; represents the vector inner product between sample i and sample j; *∈{12,21}, respectively, represents image-text and text-image:

[0109]

[0110] Where h represents the hash code of the sample, which is calculated by the sample's feature vector through the tanh activation function; the superscripts 1 and 2 represent the image modality and text modality respectively; the subscripts i and j represent the sample of one modality in the i-th sample pair and the sample of the other modality in the j-th sample pair respectively;

[0111] In order to maintain the stability, compactness and consistency of the hash code, a regularization term Q is added:

[0112]

[0113] Where n is the total number of multimodal sample pairs, a multimodal sample pair contains an image modality sample and a text modality sample, L is the length of the hash code; h is the hash code of the sample, the superscripts 1 and 2 represent the image modality and text modality respectively, and the subscript i represents the i-th sample pair.

[0114] The prototype loss function is:

[0115]

[0116] in, is the prototype loss function, is the prototype alignment loss function, is the mean square error loss function, Y' ij is the label matrix after linear change of the i-th sample pair and the j-th category, is the similarity between the image sample in the i-th sample pair and the prototype of the j-th category, is the similarity between the text sample in the i-th sample pair and the prototype of the j-th category, R ij is the similarity between the prototype of the i-th category and the prototype of the j-th category, A ij is the centralized matrix, A ij The diagonal positions are 1, the rest of the positions are -1, and the size is C*C, where C is the number of categories.

[0117] Among them, R ij Represents the similarity between the prototype of the i-th category and the prototype of the j-th category, expressed as the inner product:

[0118] R ij = <tanh(W[i,:]),tanh(W[j,:]))

[0119] Among them, W[i,:] and W[j,:] represent the prototype of the i-th category and the prototype of the j-th category respectively; tanh is the activation function.

[0120] Calculate the total loss and calculate the final loss through back propagation Gradient with respect to each parameter:

[0121]

[0122] Use Adam optimizer, add weight decay regularization, and learning rate 10 -5 , the weight decay coefficient is 10 -4 ; Set the learning rate scheduler StepLR. Starting from the first round of training, the learning rate scheduler decays the learning rate by a factor of 0.2 every 20 training cycles.

[0123] It can be understood that the final loss function of the image modality network and the text modality network is based on contrastive learning, which takes sample pairs with at least one identical category as positive sample pairs and sample pairs with completely different categories as negative sample pairs, ensuring that the model learns the positive sample pairs more accurately, while enhancing the ability to distinguish between different categories, thereby improving the accuracy of distinguishing between positive and negative samples. In addition, the positive sample pairs selected by the final loss function of the image modality network and the text modality network only focus on self-similarity, do not rely on label information, and perform stable training in the presence of noisy labels, thereby enhancing robustness and helping to better utilize the internal structure of each modality. At the same time, the final loss function of the image modality network and the text modality network takes sample pairs with completely different categories as negative sample pairs, simplifying the negative sample pair selection process and saving computing resources; better learning the boundaries between categories, reducing the interference caused by noisy labels; and better learning the characteristics of different categories;

[0124] The prototype loss function not only optimizes the similarity between positive samples and category prototypes, but also strengthens the differences between categories, allowing the model to better understand the distinction between different categories. The prototype loss function can promote effective alignment between different modalities, making the cross-modal retrieval system more stable and accurate when processing data from different modalities.

[0125] Through the consensus of image modality network, text modality network and prototype, the model's dependence on specific data distribution is reduced, and the model's generalization ability and robustness are increased; consensus learning strengthens the model's learning of category prototypes, making the representativeness of each category more prominent and the boundaries between different categories clearer.

[0126] Model evaluation can be performed by following the steps below:

[0127] S81. Load model: Read the trained model parameters and put the images, texts, and prototypes into evaluation mode.

[0128] S82. Calculate the test set hash code: Input the test set data and generate the binary hash code and corresponding label of the sample image and text;

[0129] S83. Calculate database hash code: Input database data and generate hash codes and labels of sample images and texts for retrieval and matching;

[0130] S84. Calculate cross-modal retrieval performance: Calculate the mean average precision (mAP) using both image-text and text-image retrieval methods to evaluate the retrieval performance of the model. The calculation formula is as follows:

[0131]

[0132] Among them, AP q is the average precision, which is calculated as follows:

[0133]

[0134] Among them, P q (k) is the accuracy in the first k samples of the search list, calculated as:

[0135]

[0136] Among them, Q is the total number of query samples; AP q is the average precision of the qth query; R q R is the number of relevant samples for the qth query (i.e., the total number of targets in the database that are relevant to the query); N is the total number of samples in the database (the length of the search list); δ(k) is an indicator function, where δ(k) = 1 if the kth sample is a relevant sample, otherwise δ(k) = 0;

[0137] S85. Output the final test results: Display the average accuracy results of text retrieval images and image retrieval text to measure the accuracy of cross-modal retrieval.

[0138] Step 170: Generate a cross-modal hash retrieval model based on the parameters of the image modality network, text modality network, and prototype after multiple rounds of training.

[0139] Step 180: Obtain a cross-modal search result corresponding to the cross-modal search request through a cross-modal hash search model.

[0140] Figure 5 is a module diagram of a cross-modal retrieval system based on robust double hash consensus learning according to some embodiments of this specification, such as Figure 5 As shown, the cross-modal retrieval system based on robust double hash consensus learning can include a model training module and a cross-modal retrieval module.

[0141] The model training module is used to build an image modality network, a text modality network, and a prototype; through the image modality network, the image features of the image modality samples in the multimodal sample pairs are extracted;

[0142] Extract text features of text modal samples in multimodal sample pairs through the text modal network; obtain predicted soft labels of multimodal sample pairs through the prototype; determine the prediction confidence of multimodal sample pairs based on the predicted soft labels and original labels of multimodal sample pairs; perform label correction on the predicted soft labels of multimodal sample pairs based on the prediction confidence of multimodal sample pairs; update the parameters of image modal network, text modal network and prototype using the image hash code, text hash code and corrected labels of multimodal sample pairs; generate a cross-modal hash retrieval model based on the parameters of image modal network, text modal network and prototype after multiple rounds of training;

[0143] The cross-modal retrieval module is used to obtain the cross-modal retrieval results corresponding to the cross-modal retrieval request through the cross-modal hash retrieval model.

[0144] The cross-modal retrieval system based on robust double hash consensus learning can be used to execute the cross-modal retrieval method based on robust double hash consensus learning, which will not be repeated here.

[0145] Finally, it should be understood that the embodiments described in this specification are intended only to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, by way of example and not limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly described and illustrated in this specification.

Claims

1. A cross-modal retrieval method based on robust double hash consensus learning, characterized by: include: Build image modality networks, text modality networks and prototypes; Extracting image features of image modality samples in multimodal sample pairs through image modality network; Extract text features of text modality samples in multimodal sample pairs through the text modality network; Through the prototype, we obtain the predicted soft labels of multimodal sample pairs; Determine the prediction confidence of the multimodal sample pair based on the predicted soft label and the original label of the multimodal sample pair; Based on the prediction confidence of the multimodal sample pairs, the predicted soft labels of the multimodal sample pairs are corrected; Update the parameters of the image modality network, text modality network, and prototype using the image hash code, text hash code, and rectified label of the multimodal sample pair; Generate a cross-modal hash retrieval model based on the parameters of the image modality network, text modality network, and prototype after multiple rounds of training; Through the cross-modal hash retrieval model, the cross-modal retrieval results corresponding to the cross-modal retrieval request are obtained.

2. The cross-modal retrieval method based on robust double hash consensus learning according to claim 1 is characterized in that The image modality network includes an image modality backbone network, an image modality dual-branch structure, and an image modality output layer, wherein the image modality dual-branch structure includes an image modality first branch and an image modality second branch, and the image modality first branch and the image modality second branch share a weight; The image modality backbone network is used to extract initial image features from the input image; The image modality first branch includes a multi-layer image modality first fully connected layer, and the image modality first branch is used to convert the initial image features into an image modality first embedding vector; The image modality second branch includes a multi-layer image modality second fully connected layer, and the image modality first branch is used to convert the initial image features into an image modality second embedding vector; The image modality output layer is used to normalize the image modality first embedding vector and the image modality second embedding vector to generate image features of the image modality samples in the multimodal sample pair, wherein the image features include a first image feature vector corresponding to the image modality first embedding vector and a second image feature vector corresponding to the image modality second embedding vector.

3. The cross-modal retrieval method based on robust double hash consensus learning according to claim 2 is characterized in that The text modality network includes a text modality backbone network, a text modality dual-branch structure, and a text modality output layer, wherein the text modality dual-branch structure includes a text modality first branch and a text modality second branch, and the text modality first branch and the text modality second branch share a weight; The text modality backbone network is used to convert input text data into dense vector representation; The text modality first branch includes a multi-layer text modality first fully connected layer, and the text modality first branch is used to convert the initial image features into a text modality first embedding vector; The text modality second branch includes a multi-layer image modality second fully connected layer, and the text modality first branch is used to convert the initial image features into a text modality second embedding vector; The text modality output layer is used to normalize the text modality first embedding vector and the text modality second embedding vector to generate text features of the text modality samples in the multimodal sample pair, wherein the text features include a first text feature vector corresponding to the text modality first embedding vector and a second text feature vector corresponding to the text modality second embedding vector.

4. The cross-modal retrieval method based on robust double hash consensus learning according to claim 3 is characterized in that The method of obtaining the predicted soft labels of the multimodal sample pairs through the prototype includes: generating a first feature vector group and a second feature vector group based on the first image feature vector, the second image feature vector, the first text feature vector, and the second text feature vector, wherein the first feature vector group includes one of the first image feature vector and the second image feature vector, the second feature vector group includes the other of the first image feature vector and the second image feature vector, the first feature vector group also includes one of the first text feature vector and the second text feature vector, and the second feature vector group also includes the other of the first text feature vector and the second text feature vector; Obtaining a first predicted soft label corresponding to the first feature vector group through the prototype; Obtain a second predicted soft label corresponding to the second feature vector group through the prototype; Based on the first predicted soft label and the second predicted soft label, a predicted soft label of the multimodal sample pair is obtained.

5. The cross-modal retrieval method based on robust double hash consensus learning according to claim 4 is characterized in that: The first predicted soft label corresponding to the first feature vector group is obtained through the prototype, including: S11, obtaining a predicted value based on the first feature vector group through the prototype; S12. Based on the predicted values, obtain image probability distribution and text probability distribution; S13, initialize the current number of iterations T = 1; S14. Based on the intermediate predicted soft label of the T-th iteration, the image probability distribution, and the text probability distribution, obtain the intermediate predicted soft label of the T+1-th iteration. When T=1, the intermediate predicted soft label of the T-th iteration is the original label. S15. Determine whether the iteration end condition is met. If so, complete the iteration and use the intermediate predicted soft label of the Tth iteration as the first predicted soft label corresponding to the first feature vector group. If not, set T=T+1 and execute S14.

6. The cross-modal retrieval method based on robust double hash consensus learning according to claim 4 or 5, characterized in that The step of performing label correction on the predicted soft labels of the multimodal sample pairs based on the prediction confidence of the multimodal sample pairs includes: Determining a target category index for the multimodal sample pair based on the predicted probability of each category index in the first predicted soft label and the predicted probability of each category index in the second predicted soft label; Generate the final predicted label of the multimodal sample pair based on the target category index; Based on the prediction confidence, final prediction label and original label of the multimodal sample pair, the predicted soft label of the multimodal sample pair is corrected.

7. The cross-modal retrieval method based on robust double hash consensus learning according to claim 6 is characterized in that The image hash code, text hash code and rectified label of the multimodal sample pair are used to update the parameters of the image modality network, text modality network and prototype, including: Using the cross-modal joint loss function and the prototype loss function, the parameters of the image modality network, text modality network and prototype are updated based on the image hash code, text hash code and rectified label of the multimodal sample pairs.

8. The cross-modal retrieval method based on robust double hash consensus learning according to claim 7 is characterized in that: The cross-modal joint loss function is: in, is the cross-modal joint loss function, is the similarity comparison loss function, λ is the weight, Q is the regularization term, m is the threshold, n is the number of samples, θ is the temperature coefficient, and Y ij is the label matrix of the image sample in the i-th sample pair and the text sample in the j-th sample pair, is the similarity between the image sample in the i-th sample pair and the text sample in the j-th sample pair, The similarity between the text sample in the jth sample pair and the image sample in the jth sample pair.

9. The cross-modal retrieval method based on robust double hash consensus learning according to claim 8 is characterized in that The prototype loss function is: in, is the prototype loss function, is the prototype alignment loss function, is the mean square error loss function, Y' ij is the label matrix after linear change of the i-th sample pair and the j-th category, is the similarity between the image sample in the i-th sample pair and the prototype of the j-th category, is the similarity between the text sample in the i-th sample pair and the prototype of the j-th category, R ij is the similarity between the prototype of the i-th category and the prototype of the j-th category, A ij is the centralized matrix, A ij The diagonal position is 1, the rest of the positions are -1, and the size is C*C, where C is the number of categories.

10. A cross-modal retrieval system based on robust double hash consensus learning, characterized by: Applying the cross-modal retrieval method based on robust double hash consensus learning as described in any one of claims 1 to 9, comprising: The model training module is used to build an image modality network, a text modality network, and a prototype; through the image modality network, image features of multimodal sample pairs are extracted; Extract text features of multimodal sample pairs through the text modal network; obtain predicted soft labels of multimodal sample pairs through the prototype; determine the prediction confidence of the multimodal sample pairs based on the predicted soft labels and original labels of the multimodal sample pairs; perform label correction on the predicted soft labels of the multimodal sample pairs based on the prediction confidence of the multimodal sample pairs; update the parameters of the image modal network, text modal network and prototype using the image hash code, text hash code and corrected labels of the multimodal sample pairs; generate a cross-modal hash retrieval model based on the parameters of the image modal network, text modal network and prototype after multiple rounds of training; The cross-modal retrieval module is used to obtain the cross-modal retrieval results corresponding to the cross-modal retrieval request through the cross-modal hash retrieval model.

Images