Robust cross-modal retrieval method and system based on cooperative contrast learning

Abstract

The application discloses a robust cross-modal retrieval method and system based on collaborative contrast learning, and relates to the technical field of artificial intelligence. The method comprises the following steps: initializing two models with the same structure and independent parameters; inputting an image-text sample pair in the current batch into the two models for double-path forward inference to obtain image features, text features, classification prediction probability distribution, pseudo-hard labels and confidence; cross-purification screening is performed based on the confidence of the two models; and a training set is constructed by using high-confidence samples. Joint loss is calculated on the two training sets. The two models are optimized by using an asynchronous parameter updating strategy to complete the training of the cross-modal retrieval model. Through the double-model collaboration and cross-purification, the application blocks the propagation of noise labels, suppresses confirmation bias, enhances the robustness of features by combining contrast and classification learning, reduces the dependence on artificial labeling, and improves the retrieval accuracy and generalization ability in the label noise scene.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and specifically to a robust cross-modal retrieval method and system based on collaborative contrastive learning. Background Technology

[0002] Cross-modal retrieval, belonging to the field of multimodal data analysis, aims to establish semantic relationships between different modalities of data, such as images and text, enabling the retrieval of related data from one modality based on input from another. With the increasing scale of multimodal data on the internet, this technology has become an important tool for information acquisition. In practical applications, fine-grained annotation of large-scale image-text pairs is costly, and automated annotation tools generally have accuracy limitations, inevitably leading to label noise in training datasets.

[0003] Traditional cross-modal retrieval methods typically rely on the assumption of perfectly accurate training data labels. They map features from different modalities to a unified space using contrastive learning or classification loss to achieve semantic alignment. However, when mislabeled training data exists, erroneous supervision signals can mislead the model's optimization direction, causing a shift in the feature space and impacting retrieval accuracy. To address noise issues, some solutions introduce single-model-based sample selection mechanisms, filtering low-confidence samples using preset thresholds. These methods are prone to confirmation bias; once the model makes an incorrect high-confidence prediction for a noisy sample, it reinforces this error during training, failing to self-correct. Furthermore, existing robust learning methods often focus on noise handling within a single modality, lacking effective strategies to suppress noise propagation during cross-modal interactions. This makes it difficult to fully exploit complementary information between modalities while suppressing noise interference. Therefore, balancing noise suppression with cross-modal semantic association learning in environments with unreliable labels is a major bottleneck for this type of technology. Summary of the Invention

[0004] This invention provides a robust cross-modal retrieval method and system based on collaborative contrastive learning, which solves the technical problems of existing cross-modal retrieval methods being easily misled by erroneous supervision signals when the training data contains label noise, and causing feature space shift due to confirmation bias, thereby resulting in decreased retrieval accuracy and limited generalization ability.

[0005] This invention is achieved through the following technical solution:

[0006] Firstly, this application provides a robust cross-modal retrieval method based on collaborative contrastive learning, comprising the following steps:

[0007] Initialize two models with identical structures and independent parameters, named Model A and Model B.

[0008] Input the current batch of image-text sample pairs into Model A and Model B respectively, and obtain the image features, text features, classification prediction probability distribution, pseudo hard label and confidence of each sample pair through dual-path forward inference;

[0009] Based on the confidence scores of Model A and Model B, cross-filtering is performed to obtain the training set for Model A. and the training set of Model B ,in It consists of samples whose confidence level for model A is greater than a preset confidence threshold. It consists of samples whose confidence level for Model B is greater than the preset confidence threshold;

[0010] In respectively and The joint loss of model A and model B is calculated, which includes classification loss and contrast loss. The classification loss is calculated based on the pseudo hard labels of the other model, and the contrast loss is the average of image-text contrast loss and text-image contrast loss.

[0011] Asynchronous parameter updates are performed on model A and model B respectively to complete the training of the cross-modal retrieval model.

[0012] A further optimized solution is that the asynchronous parameter update steps specifically include:

[0013] Calculate the gradient of the total loss of model A with respect to all parameters of model A, update only the parameters of model A, and keep the parameters of model B unchanged;

[0014] Calculate the gradient of the total loss of model B with respect to all parameters of model B, update only the parameters of model B, and keep the parameters of model A unchanged;

[0015] The asynchronous update maintains the parameter differences between model A and model B, enabling diversity in collaborative learning.

[0016] A further optimization is that the classification loss is calculated as follows:

[0017] For the classification loss of model A, in the training set Above, using the pseudo-hard labels generated by model B as the supervision signal, the cross-entropy loss function is used for calculation, and the formula is as follows:

[0018] ;

[0019] in, It is a set Size, It is the probability value of the pseudo hard label of model B in the predicted probability distribution of model A;

[0020] For the classification loss of model B, in the training set Above, using the pseudo-hard labels generated by model A as the supervision signal, the cross-entropy loss function is used for calculation, and the formula is as follows:

[0021] ;

[0022] in, It is a set Size, It is the probability value corresponding to the pseudo hard label of model A in the predicted probability distribution of model B.

[0023] A further optimization scheme is as follows: For the contrast loss of model A, the image-text contrast loss formula is:

[0024] ;

[0025] The formula for text-image contrast loss is:

[0026] ;

[0027] The total contrastive loss of model A is the mean of the two:

[0028] ;

[0029] in, , These are the image features and text features extracted from the j-th sample by model A, respectively. Here, is the temperature hyperparameter, and sim represents the cosine similarity.

[0030] A further optimized solution is that the joint loss is calculated as follows:

[0031] The total loss of model A is the weighted sum of the classification loss and the contrastive loss:

[0032] ;

[0033] The total loss of model B is the weighted sum of the classification loss and the contrastive loss:

[0034] ;

[0035] Where λ is a hyperparameter that balances classification loss and contrast loss.

[0036] A further optimized solution is that the step of obtaining the pseudo-hard label through dual-path forward inference specifically includes:

[0037] For model A, the category index corresponding to the highest probability in the classification prediction probability distribution is used as the pseudo hard label:

[0038] ;

[0039] For model B, the category index corresponding to the highest probability in the classification prediction probability distribution is used as the pseudo hard label:

[0040] ;

[0041] in, , These are the predicted probabilities of model A and model B for sample j belonging to category c, respectively.

[0042] A further optimization is that the confidence level is calculated as follows:

[0043] The confidence score of model A is the maximum probability value in the classification prediction probability distribution:

[0044] ;

[0045] The confidence score of model B is the maximum probability value in the classification prediction probability distribution:

[0046] ;

[0047] in, , As a basis for cross-cleaning screening, only when the confidence level is greater than the preset confidence threshold. At that time, the sample is included in the other party's training set.

[0048] A further optimization scheme is as follows: the image features are extracted through an image encoder, which is constructed using a convolutional neural network or a visual Transformer; the text features are extracted through a text encoder, which is constructed using a BERT series model.

[0049] A further optimization plan includes, after training the cross-modal retrieval model, the following:

[0050] In response to receiving a user's search request, extract the feature vector of the query data;

[0051] Calculate the similarity between the feature vector of the queried data and the feature vector pre-stored in the database;

[0052] The search results are returned in a list sorted by similarity.

[0053] Secondly, this application provides a robust cross-modal retrieval system based on collaborative contrastive learning, comprising:

[0054] The model initialization module is used to initialize two models with identical structures and independent parameters, namely model A and model B.

[0055] The forward inference module is used to input the current batch of image-text sample pairs into Model A and Model B respectively, and obtain the image features, text features, classification prediction probability distribution, pseudo hard label and confidence of each sample pair through dual-path forward inference.

[0056] The cross-cleaning module is used to perform cross-cleaning and filtering based on the confidence scores of model A and model B, respectively, to obtain the training set of model A. and the training set of Model B ,in It consists of samples whose confidence level for model A is greater than a preset confidence threshold. It consists of samples whose confidence level for Model B is greater than the preset confidence threshold;

[0057] The loss calculation module is used to calculate the loss in the following ways: and The joint loss of model A and model B is calculated, which includes classification loss and contrast loss. The classification loss is calculated based on the pseudo hard labels of the other model, and the contrast loss is the average of image-text contrast loss and text-image contrast loss.

[0058] The parameter update module is used to asynchronously update the parameters of model A and model B respectively, thereby completing the training of the cross-modal retrieval model.

[0059] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0060] By using a dual-model collaboration and cross-cleaning screening mechanism, the propagation path of erroneous supervision signals generated by noise labels is effectively blocked during the training process, the confirmation bias that is easily generated during single model training is suppressed, and the feature learning accuracy and anti-interference ability of the model in the environment of label noise are significantly improved.

[0061] By introducing a contrast loss that does not rely on manual annotation and a classification loss based on the high confidence pseudo-label of the other party for joint optimization, the semantic discriminativeness and representational robustness of cross-modal features are simultaneously enhanced on the purified subset, ensuring the correct alignment of image and text features in space.

[0062] The asynchronous parameter update strategy maintains the difference between the two model learning paths, avoids model collapse, makes the training process more stable and efficient, and reduces the dependence on large-scale, high-quality manually labeled data. As a result, it achieves higher retrieval accuracy and stronger generalization ability on cross-modal retrieval datasets with mislabeled data. Attached Figure Description

[0063] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0064] Figure 1 A flowchart of a robust cross-modal retrieval method based on collaborative contrastive learning provided in this application embodiment;

[0065] Figure 2 This is a functional block diagram of a robust cross-modal retrieval system based on collaborative contrastive learning, provided in an embodiment of this application. Detailed implementation details.

[0066] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0067] Firstly, such as Figure 1 As shown, this application provides a robust cross-modal retrieval method based on collaborative contrastive learning, including the following steps:

[0068] Step S1: Initialize two models with identical structures and independent parameters, namely model A and model B;

[0069] Step S2: Input the current batch of image-text sample pairs into Model A and Model B respectively, and obtain the image features, text features, classification prediction probability distribution, pseudo hard label and confidence level of each sample pair through dual-path forward inference;

[0070] Step S3: Based on the confidence scores of Model A and Model B, perform cross-filtering to obtain the training set of Model A. and the training set of Model B ,in It consists of samples whose confidence level for model A is greater than a preset confidence threshold. It consists of samples whose confidence level for Model B is greater than the preset confidence threshold;

[0071] Step S4: In respectively and The joint loss of model A and model B is calculated, which includes classification loss and contrast loss. The classification loss is calculated based on the pseudo hard labels of the other model, and the contrast loss is the average of image-text contrast loss and text-image contrast loss.

[0072] Step S5: Asynchronously update the parameters of Model A and Model B respectively to complete the training of the cross-modal retrieval model.

[0073] This embodiment constructs a dual-model collaborative architecture with structurally similar but parameter-decoupled features. It utilizes the natural differences introduced by random initialization to simulate ensemble learning effects. A cross-purification and filtering mechanism ensures that each model only absorbs the pseudo-hard label supervision signal from the other model at high confidence levels, effectively blocking the accumulation path of confirmation bias caused by noisy labels in a single model. Simultaneously, it combines contrastive learning loss that does not rely on manual annotation, enhancing cross-modal feature alignment while mining the inherent geometric structure information of the data. Coupled with dynamic course learning based on confidence thresholds and a strict asynchronous parameter update strategy, it ensures both the purity of the supervision signal and the stability of the training process, while maintaining the diversity between the two models to avoid collaborative failure. This significantly improves the robustness, feature discriminativeness, and generalization ability of the cross-modal retrieval model in label-noisy environments, reduces the cost of relying on high-quality labeled data, and provides reliable technical support for accurate cross-modal information acquisition in practical applications.

[0074] In one embodiment, step S1: Initialize two models with identical structures and independent parameters, namely model A and model B, specifically including the following steps:

[0075] Step S11: Based on the predefined cross-modal feature extraction and classification architecture, the parameter sets of model A and model B are instantiated independently to generate two sets of initial weights that are not shared between them.

[0076] Specifically, the parameter set of model A is defined as follows:

[0077] ;

[0078] Define the parameter set of model B as follows:

[0079] ;

[0080] in, and The image feature extractor weights correspond to those of Model A and Model B, respectively. and These correspond to the text feature extractor weights of Model A and Model B, respectively. and These correspond to the classifier weights for model A and model B, respectively.

[0081] Step S12: Based on the preset random number seed strategy, randomly assign values ​​to the tensor values ​​in the two sets of parameters to establish the initial state of training model A and model B.

[0082] Specifically, a randomization method following a specific probability distribution is used to fill the gaps. and Each parameter matrix ensures that model A and model B have different convergence paths at the beginning of training.

[0083] Step S13: Configure the batch image-text sample input interface to establish a mapping link from raw pixels and word embeddings to deep semantic features.

[0084] Specifically, the N image-text pairs contained in the current batch Import the two independent model instances, Model A and Model B, respectively. The image data representing the j-th sample, The text data representing the j-th sample. Model A uses an image feature extraction network. Text feature extraction network Transform the input into an image feature vector With text feature vectors ;

[0085] Model B performs parallel feature extraction operations to generate corresponding image feature vectors. With text feature vectors .

[0086] Step S14: Use a fully connected classification layer to perform semantic space projection on the fused cross-modal features, and output the probability distribution of each category and the corresponding deterministic score.

[0087] Specifically, model A will concatenate the features Input classifier The predicted probability distribution is obtained. ; and through calculation Identify pseudo hard tags through calculation Determine the confidence level. Model B follows the same processing logic and outputs the predicted probability distribution. Pseudo-hard labels and confidence level This provides a quantitative basis for the subsequent cross-purification screening of Model A and Model B.

[0088] This embodiment constructs a dual-model architecture with the same structure but decoupled parameters, breaking the path of confirmation error accumulation of a single model from the source. It uses the differences generated by random initialization to simulate the effect of ensemble learning, enabling model A and model B to explore the feature space from different optimization starting points. In subsequent iterations, the cross-supervision mechanism is used to achieve effective filtering of noise labels and collaborative extraction of robust features.

[0089] In one embodiment, step S2: Input the current batch of image-text sample pairs into model A and model B respectively, and obtain the image features, text features, classification prediction probability distribution, pseudo hard labels and confidence scores of each sample pair through dual-path forward inference. Specifically, this includes the following steps:

[0090] Step S21: Use the N image-text pairs collected in the current batch as the excitation source and synchronously send them to the embedding layers of Model A and Model B to trigger the feature encoding process of Model A and Model B in parallel, and produce a high-dimensional semantic representation vector.

[0091] Specifically, for the j-th sample pair within a batch, the image data it contains is denoted as... Text data is denoted as Model A calls the image feature extraction network. Text feature extraction network Perform mapping operations respectively and Thus, the image feature vector of model A is obtained. With text feature vectors Model B performs parallel feature extraction operations, generating image feature vectors corresponding to Model B. With text feature vectors .

[0092] Step S22: Fuse the image and text features extracted by the same model, perform nonlinear transformation through a fully connected classifier, map the feature spaces of model A and model B to a predefined semantic category space, and output a normalized probability distribution matrix.

[0093] Specifically, model A will concatenate the cross-modal features Input classifier g, calculate the predicted probability distribution , where vector elements The representation model A determines the probability that sample pair j belongs to category c, and the sum of all elements satisfies Model B follows the same mapping logic, and outputs the predicted probability distribution corresponding to Model B. .

[0094] Step S23: Traverse the probability distribution vectors output by Model A and Model B, lock the category index with the highest probability magnitude, and establish it as the deterministic category of the sample from the perspective of the current model.

[0095] Specifically, Model A performs calculations. Obtain the pseudo-hard label for sample j. This operation returns the class index c that maximizes the predicted probability of model A. Model B uses symmetric operations. The pseudo hard labels corresponding to model B are generated, and this result provides a discrete classification basis for the subsequent cross-supervision between model A and model B.

[0096] Step S24: Evaluate the confidence of Model A and Model B in their output decisions, extract the peak value in the probability distribution as the confidence score, and use this to measure the reliability of the current prediction results.

[0097] Specifically, Model A calculates... The confidence score of model A is obtained, which reflects the highest level of confidence model A has in classifying sample j. Model B is calculated... Obtain the confidence index of model B.

[0098] Among them, the confidence level of model A confidence level with model B It will be used as the core quantitative evidence for sample selection and construction of personalized training subsets of Model A and Model B in step S3.

[0099] This embodiment, through a dual-path independent forward inference architecture of Model A and Model B, not only achieves parallel generation of cross-modal features and classification prediction, but more importantly, it uses probabilistic statistical methods to quantify the decision boundaries and confidence levels of Model A and Model B. This provides an accurate and measurable data foundation for the subsequent cross-purification and screening mechanism of Model A and Model B, effectively supporting the removal of false information from supervisory signals in noisy environments.

[0100] In one embodiment, step S3: Based on the confidence scores of model A and model B, perform cross-filtering to obtain the training set D of model A. A and the training set D of model B B D A D consists of samples whose confidence level for model A is greater than a preset confidence threshold. B The sample consists of samples whose confidence level in model B is greater than a preset confidence threshold, and specifically includes the following steps:

[0101] Step S31: Traverse all sample indices in the current batch, compare the confidence values ​​output by model A and model B with the preset confidence threshold one by one, and identify the sample ranges that model A and model B determine as having high certainty.

[0102] Specifically, the confidence threshold is set as follows: This is used to define the confidence boundary of a sample. For the j-th sample in a batch, the confidence score of model B is extracted. and Compare them.

[0103] If satisfied > If the conditions are met, then model B is deemed to have high reliability in predicting the sample, and the sample index is included in the candidate training set of model A.

[0104] Step S32: Based on the high confidence screening results of model B, aggregate the sample indexes that meet the conditions, generate a training subset specifically for optimizing the parameters of model A, and establish the learning materials for model A in this round.

[0105] Specifically, the training subset of model A is defined as

[0106] ;

[0107] Where N represents the total number of samples in the current batch, and j represents the sample index. This represents the confidence level of model B's prediction for sample j. This set... In essence, it consists of high-confidence knowledge points identified by Model B, providing Model A with a source of supervisory signals that has undergone preliminary purification.

[0108] Step S33: Based on the high confidence screening results of model A, aggregate the sample indexes that meet the conditions, generate a training subset specifically for optimizing the parameters of model B, and establish the learning materials for model B in this round.

[0109] Specifically, the training subset of model B is defined as:

[0110] ;

[0111] in, This represents the confidence level of model A's prediction for sample j. This set... The model consists of highly confident knowledge points identified by model A, ensuring that model B primarily receives reliable supervision information from model A during training.

[0112] Step S34: Monitor the training subset D of model A A With model B's training subset D B As the scale of the iteration process evolves, the difficulty level of the samples participating in the training is adaptively adjusted to achieve a smooth transition from simple samples to complex samples.

[0113] Specifically, in the initial training phase, because model A and model B have a low degree of fit, only a very small number of samples have confidence scores exceeding the threshold τconf. At this time, the training subset D of model A... A With model B's training subset D B It mainly consists of simple samples with obvious features. As the number of iterations increases and the performance of models A and B improves, more difficult samples with marginally increasing confidence are gradually included in the training subset D of model A. A With model B's training subset DB This dynamic screening mechanism, based on confidence thresholds, can automatically arrange courses from easy to difficult without human intervention.

[0114] This embodiment utilizes a cross-purification screening mechanism between Model A and Model B, employing a collaborative model where both models act as teachers to effectively block the accumulation path of confirmation bias caused by noise labels in a single model. Simultaneously, it achieves dynamic grading of training samples by leveraging confidence thresholds. This not only ensures the purity of the supervision signal but also conforms to the cognitive laws of model learning, significantly improving the stability and noise resistance of the training process of Model A and Model B.

[0115] In one embodiment, step S4: respectively in D A and D B The joint loss of model A and model B is calculated, which includes classification loss and contrast loss. The classification loss is calculated based on the pseudo-hard labels of the other model, and the contrast loss is the mean of image-text contrast loss and text-image contrast loss. The specific steps include:

[0116] Step S41: In the training subset D of model A A With model B's training subset D B Within this model, the cross-entropy error between the current model's predicted distribution and the pseudo-hard labels provided by the collaborative model is calculated, forcing Model A and Model B to align with the decision boundary of their high-confidence peer.

[0117] Specifically, for model A, in its training subset D A The classification loss is calculated as follows:

[0118] ;

[0119] Among them, |D A | Represents the cardinality of the set, used for loss normalization. In the predicted probability distribution of model A, the pseudo-hard label corresponding to model B. The probability value of the category. This operation aims to maximize the predicted probability of model A in the correct category identified by model B, achieving one-way distillation of knowledge. The classification loss of model B. Constructed following symmetric logic, utilizing the pseudo-hard labels of model A. The parameter optimization for model B is calculated using the following formula:

[0120] ;

[0121] Step S42: Utilize the similarity relationship between positive and negative sample pairs within a batch to construct a comparative learning objective between Model A and Model B, thereby encouraging matched image-text pairs to attract each other in the feature space while pushing away unmatched pairs.

[0122] Specifically, the image-to-text contrast loss of model A. The definition is as follows:

[0123] ;

[0124] Where N is the batch size, Represents the cosine similarity operator. Temperature hyperparameters are used to adjust the distribution sharpness. The molecular component measures anchor point image features. Its paired text features The similarity is calculated by summing the logarithmic similarities between the anchor feature and all text features within the batch in the denominator. This loss function does not rely on manual annotation; it strengthens the cross-modal semantic alignment of model A solely through the pairing relationships within the data itself.

[0125] Step S43: Perform symmetrical similarity measurement in opposite directions to ensure that the text features of Model A and Model B can effectively retrieve the corresponding visual representations in the projection space, thus improving bidirectional cross-modal association.

[0126] Specifically, the text-to-image contrast loss of model A The definition is as follows:

[0127] ;

[0128] Among them, anchor text features Its paired image features Forming positive sample pairs, along with other image features within the batch. This forms a negative sample pair. The contrastive loss component of Model B. and Constructed using the exact same paradigm, but based on the feature vectors extracted by model B. and Perform the calculation.

[0129] Step S44: Aggregate the comparison errors of Model A and Model B in both directions to form a unified representation learning objective, ensuring that Model A and Model B have balanced bidirectional retrieval capabilities.

[0130] Specifically, the total contrast loss of model A Obtained by taking the arithmetic mean:

[0131] ;

[0132] This fusion strategy eliminates modality bias that may result from unidirectional optimization, forcing Model A and Model B to maintain consistent performance across both image retrieval of text and text retrieval of images. The total contrastive loss of Model B... It is also obtained by fusing its component loss in the same way.

[0133] Step S45: Introduce a balancing coefficient to weight and superimpose the classification supervision signal and the contrastive self-supervision signal, and output the final composite gradient descent target of model A and model B.

[0134] Specifically, the total loss of model A The calculation is as follows:

[0135] ;

[0136] Where λ is a hyperparameter that balances classification accuracy and representation quality. The total loss of Model B... The calculation is as follows:

[0137] ;

[0138] These two total loss functions serve as the driving sources for asynchronous backpropagation of Model A and Model B, respectively. They include both cleaned semantic label information from collaborating partners and retain the inherent structured constraints of the data, providing a precise optimization direction for the subsequent independent updates of the parameters of Model A and Model B.

[0139] This embodiment achieves the dual objectives of classification and comparison by jointly optimizing Model A and Model B. While using cross-label cleaning to suppress noise interference, it further mines the geometric structure information inside the data through contrastive learning. This not only strengthens the discriminative power of cross-modal features, but also significantly enhances the robustness and generalization ability of the feature spaces of Model A and Model B, enabling Model A and Model B to learn accurate semantic alignment relationships even in low-quality labeled environments.

[0140] In one embodiment, step S5: asynchronously update the parameters of model A and model B respectively to complete the training of the cross-modal retrieval model, specifically including the following steps:

[0141] Step S51: Based on the total loss function of model A, solve for the set of partial derivatives of model A with respect to all network parameters of model A, and establish the optimal descent trajectory in the parameter space of model A.

[0142] Specifically, calculate the loss function. Relative to the parameter set of model A gradient This gradient vector contains information about the contribution of each parameter of model A to the current prediction error.

[0143] Step S52: Based on the calculated gradient information and the preset learning rate, perform a correction operation on the parameter set of model A to make model A evolve in the direction of minimizing loss.

[0144] Specifically, the parameters of model A are updated using the gradient descent algorithm or its adaptive variant:

[0145] ;

[0146] Where η represents the learning rate that controls the step size of the parameter update. This operation only modifies the parameters of model A itself. without affecting the parameters of model B. This ensures that model A completes an independent knowledge internalization process within the current training step.

[0147] Step S53: Independent of the optimization path of model A, based on the total loss function of model B, solve for the set of partial derivatives of model B with respect to all network parameters of model B, and establish another independent parameter optimization trajectory in the parameter space of model B.

[0148] Specifically, calculate the loss function. Relative to the parameter set of model B gradient The gradient vector reflects model B on its purified training subset D. B Error distribution on.

[0149] Step S54: Based on the gradient feedback unique to Model B, perform independent correction operations on the parameter set of Model B to maintain the difference between Model B and Model A in the parameter space.

[0150] Specifically, the parameters of model B are updated using the gradient descent algorithm:

[0151] ;

[0152] Here, η represents the learning rate. This step strictly isolates the parameter flow between model A and model B, preventing model degradation caused by parameter sharing or synchronous updates, and ensuring the continued effectiveness of the collaborative learning mechanism between model A and model B.

[0153] Repeat the complete closed-loop process of dual-path forward inference, cross-filtering, joint loss optimization, and asynchronous parameter updates for both Model A and Model B until Model A and Model B reach the predetermined number of iterations or their performance metrics stabilize.

[0154] Specifically, after each iteration, both Model A and Model B incorporate the cleansing supervision signals provided by each other and optimize their feature spaces under the constraints of contrastive learning. As the number of iterations increases, the classification accuracy and cross-modal retrieval precision of Model A and Model B in noisy environments continue to improve, ultimately outputting a pair of robust cross-modal retrieval models for practical image-text matching tasks.

[0155] This embodiment effectively blocks parameter coupling between model A and model B through a strict asynchronous parameter update mechanism, maintains the model diversity required for collaborative learning, avoids collaborative failure caused by convergence of model A and model B, and ensures that in a noisy label environment, model A and model B can continuously provide reliable supervision signals to each other from different perspectives, ultimately achieving efficient convergence and excellent generalization performance of the cross-modal retrieval model.

[0156] Secondly, such as Figure 2 As shown, this application provides a robust cross-modal retrieval system based on collaborative contrastive learning, including a model initialization module 100, a forward inference module 200, a cross-cleaning module 300, a loss calculation module 400, and a parameter update module 500.

[0157] The model initialization module 100 is used to initialize two models with the same structure but independent parameters, namely model A and model B.

[0158] The forward inference module 200 is used to input the current batch of image-text sample pairs into the model A and model B respectively, and obtain the image features, text features, classification prediction probability distribution, pseudo hard label and confidence of each sample pair through dual-path forward inference;

[0159] The cross-cleaning module 300 is used to perform cross-cleaning and filtering based on the confidence scores of model A and model B, respectively, to obtain the training set of model A. and the training set of Model B ,in It consists of samples whose confidence level for model A is greater than a preset confidence threshold. It consists of samples whose confidence level for Model B is greater than the preset confidence threshold;

[0160] The loss calculation module 400 is used to calculate the loss respectively in and The joint loss of model A and model B is calculated, which includes classification loss and contrast loss. The classification loss is calculated based on the pseudo hard labels of the other model, and the contrast loss is the average of image-text contrast loss and text-image contrast loss.

[0161] The parameter update module 500 is used to asynchronously update the parameters of model A and model B respectively, thereby completing the training of the cross-modal retrieval model.

[0162] The functional implementation of each module in the robust cross-modal retrieval device based on collaborative contrastive learning corresponds to each step in the embodiment of the robust cross-modal retrieval method based on collaborative contrastive learning. Their functions and implementation processes will not be described in detail here.

[0163] Thirdly, embodiments of this application provide a robust cross-modal retrieval device based on collaborative contrastive learning. The robust cross-modal retrieval device based on collaborative contrastive learning can be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0164] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0165] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within the robust cross-modal retrieval device based on cooperative contrastive learning, as well as interfaces used for interconnecting the robust cross-modal retrieval device based on cooperative contrastive learning with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0166] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0167] The processor can be a general-purpose processor, which can call a robust cross-modal retrieval program based on cooperative contrastive learning stored in memory and execute the robust cross-modal retrieval method based on cooperative contrastive learning provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the robust cross-modal retrieval program based on cooperative contrastive learning is called can be referred to in the various embodiments of the robust cross-modal retrieval method based on cooperative contrastive learning in this application, and will not be repeated here.

[0168] Fourthly, embodiments of this application also provide a readable storage medium.

[0169] This application stores a robust cross-modal retrieval program based on cooperative contrastive learning on a readable storage medium, wherein when the robust cross-modal retrieval program based on cooperative contrastive learning is executed by a processor, it implements the steps of the robust cross-modal retrieval method based on cooperative contrastive learning as described above.

[0170] The method implemented when the robust cross-modal retrieval program based on collaborative contrastive learning is executed can be referred to in the various embodiments of the robust cross-modal retrieval method based on collaborative contrastive learning in this application, and will not be repeated here.

[0171] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0172] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A robust cross-modal retrieval method based on collaborative contrastive learning, characterized in that, Includes the following steps: Initialize two models with identical structures and independent parameters, named Model A and Model B. Input the current batch of image-text sample pairs into Model A and Model B respectively, and obtain the image features, text features, classification prediction probability distribution, pseudo hard label and confidence of each sample pair through dual-path forward inference; Based on the confidence scores of Model A and Model B, cross-filtering is performed to obtain the training set for Model A. and the training set of Model B ,in It consists of samples whose confidence level for model A is greater than a preset confidence threshold. It consists of samples whose confidence level for Model B is greater than the preset confidence threshold; In respectively and The joint loss of model A and model B is calculated, which includes classification loss and contrast loss. The classification loss is calculated based on the pseudo hard labels of the other model, and the contrast loss is the average of image-text contrast loss and text-image contrast loss. Asynchronous parameter updates are performed on model A and model B respectively to complete the training of the cross-modal retrieval model.

2. The robust cross-modal retrieval method based on collaborative contrastive learning according to claim 1, characterized in that, The asynchronous parameter update steps specifically include: Calculate the gradient of the total loss of model A with respect to all parameters of model A, update only the parameters of model A, and keep the parameters of model B unchanged; Calculate the gradient of the total loss of model B with respect to all parameters of model B, update only the parameters of model B, and keep the parameters of model A unchanged; The asynchronous update maintains the parameter differences between model A and model B, enabling diversity in collaborative learning.

3. The robust cross-modal retrieval method based on collaborative contrastive learning according to claim 1, characterized in that, The classification loss is calculated as follows: For the classification loss of model A, in the training set Above, using the pseudo-hard labels generated by model B as the supervision signal, the cross-entropy loss function is used for calculation, and the formula is as follows: ； in, It is a set Size, It is the probability value of the pseudo hard label of model B in the predicted probability distribution of model A; For the classification loss of model B, on the training set Above, using the pseudo-hard labels generated by model A as the supervision signal, the cross-entropy loss function is used for calculation, and the formula is as follows: ； in, It is a set Size, It is the probability value corresponding to the pseudo hard label of model A in the predicted probability distribution of model B.

4. The robust cross-modal retrieval method based on collaborative contrastive learning according to claim 1, characterized in that, For the contrast loss of model A, the formula for image-text contrast loss is: ； The formula for text-image contrast loss is: ； The total contrastive loss of model A is the mean of the two: ； in, , These are the image features and text features extracted from the j-th sample by model A, respectively. Here, is the temperature hyperparameter, and sim represents the cosine similarity.

5. The robust cross-modal retrieval method based on collaborative contrastive learning according to claim 1, characterized in that, The joint loss is calculated as follows: The total loss of model A is the weighted sum of the classification loss and the contrastive loss: ； The total loss of model B is the weighted sum of the classification loss and the contrastive loss: ； Where λ is a hyperparameter that balances classification loss and contrast loss.

6. The robust cross-modal retrieval method based on collaborative contrastive learning according to claim 1, characterized in that, The steps for obtaining pseudo-hard labels through dual-path forward inference specifically include: For model A, the category index corresponding to the highest probability in the classification prediction probability distribution is used as the pseudo hard label: ； For model B, the category index corresponding to the highest probability in the classification prediction probability distribution is used as the pseudo hard label: ； in, , These are the predicted probabilities of model A and model B for sample j belonging to category c, respectively.

7. The robust cross-modal retrieval method based on collaborative contrastive learning according to claim 1, characterized in that, The confidence level is calculated as follows: The confidence score of model A is the maximum probability value in the classification prediction probability distribution: ； The confidence score of model B is the maximum probability value in the classification prediction probability distribution: ； in, , As a basis for cross-cleaning screening, only when the confidence level is greater than the preset confidence threshold. At that time, the sample is included in the other party's training set.

8. The robust cross-modal retrieval method based on collaborative contrastive learning according to claim 1, characterized in that, The image features are extracted by an image encoder, which is constructed using a convolutional neural network or a visual Transformer; the text features are extracted by a text encoder, which is constructed using a BERT series model.

9. The robust cross-modal retrieval method based on collaborative contrastive learning according to claim 1, characterized in that, After training the cross-modal retrieval model, the following steps are also included: In response to receiving a user's search request, extract the feature vector of the query data; Calculate the similarity between the feature vector of the queried data and the feature vector pre-stored in the database; The search results are returned in a list sorted by similarity.

10. A robust cross-modal retrieval system based on collaborative contrastive learning, characterized in that, include: The model initialization module is used to initialize two models with identical structures and independent parameters, namely model A and model B. The forward inference module is used to input the current batch of image-text sample pairs into Model A and Model B respectively, and obtain the image features, text features, classification prediction probability distribution, pseudo hard label and confidence of each sample pair through dual-path forward inference. The cross-cleaning module is used to perform cross-cleaning and filtering based on the confidence scores of model A and model B, respectively, to obtain the training set of model A. and the training set of Model B ,in It consists of samples whose confidence level for model A is greater than a preset confidence threshold. It consists of samples whose confidence level for Model B is greater than the preset confidence threshold; The loss calculation module is used to calculate the loss in the following ways: and The joint loss of model A and model B is calculated, which includes classification loss and contrast loss. The classification loss is calculated based on the pseudo hard labels of the other model, and the contrast loss is the average of image-text contrast loss and text-image contrast loss. The parameter update module is used to asynchronously update the parameters of model A and model B respectively, thus completing the training of the cross-modal retrieval model.

Images