Robust cross-modal retrieval method based on noise tag rematching and pseudo classification
By employing noise label re-matching and pseudo-classification methods, the robustness of the cross-modal retrieval model was improved, the impact of noisy data on model performance was addressed, and retrieval accuracy and stability were enhanced.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING NORMAL UNIVERSITY
- Filing Date
- 2026-01-20
- Publication Date
- 2026-05-01
AI Technical Summary
Existing technologies in cross-modal matching suffer from noise data, making it difficult for models to adaptively correct erroneous image-text pairs. This affects the model's discriminative learning ability and retrieval accuracy, and the system cannot maintain robustness and stability in complex noisy scenarios.
By employing noise label rematching and pseudo-classification methods, a pre-trained model is used to detect and correct text labels for noisy images. A knowledge base is constructed to evaluate the performance impact of training samples, generate confidence weights, and combine pseudo-classification tasks to optimize the model, thus forming a robust cross-modal retrieval method.
It improves the annotation quality and retrieval accuracy of cross-modal datasets, prevents noisy data from misleading model parameters, and enhances the stability and generalization ability of the model in complex noisy scenarios.
Smart Images
Figure CN121958596A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a robust cross-modal retrieval method based on noise label rematching and pseudo-classification. Background Technology
[0002] Cross-modal matching technology aims to establish semantic correspondences between different modalities of data, such as images and text, and is the foundation for artificial intelligence applications such as image description generation, cross-modal retrieval, and visual question answering. With the evolution of deep learning technology, visual language models based on large-scale pre-trained data have effectively improved cross-modal semantic alignment capabilities. These techniques typically utilize massive amounts of image and text data for joint training, mapping them to a common feature space to reduce semantic differences between modalities, thereby enabling the understanding and interaction of complex multimodal data.
[0003] Existing technologies often rely on large-scale datasets crawled from the web, which inevitably contain a large number of noisy correspondences. Models struggle to adaptively correct erroneous image-text pairs in the feature space and are prone to fitting noisy samples during training, leading to deep neural networks remembering incorrect associations. This not only inhibits the model's discriminative learning ability on clean samples but also causes a decrease in retrieval accuracy, making it impossible to maintain the robustness and stability of the system in complex noisy scenarios. Summary of the Invention
[0004] To overcome the above shortcomings, this invention provides a robust cross-modal retrieval method based on noise label re-matching and pseudo-classification, aiming to improve the problem of model instability.
[0005] In a first aspect, the present invention provides the following technical solution: a robust cross-modal retrieval method based on noise label re-matching and pseudo-classification, comprising: Step S1: Obtain a cross-modal training set containing image data and text data, and extract image features and text features respectively; Step S2: Based on the sample loss distribution characteristics, the cross-modal training set is divided into a clean set and a noisy set, and samples are selected from the clean set to construct a knowledge base for evaluating model performance; Step S3: Use a pre-trained model to perform semantic consistency detection on the noisy images in the noise set. When the preset improvement conditions are met, correct the text labels corresponding to the noisy images to obtain the optimized sample correspondence. Step S4: Use the knowledge base to evaluate the impact of training samples on model performance, and generate confidence weights for each training sample based on the evaluation results; Step S5: Construct a joint optimization objective that includes the pseudo-classification task, and retrain the model using the confidence weights to obtain the final cross-modal retrieval model.
[0006] Preferably, in step S2, the step of dividing the cross-modal training set into a clean set and a noisy set based on the sample loss distribution characteristics includes: Calculate the cross-entropy loss value for each sample in the training set; The distribution of the cross-entropy loss value is modeled using a Gaussian mixture model, and the posterior probability that each sample belongs to a clean sample is calculated. Samples with a posterior probability greater than a preset threshold are assigned to the clean set, and the remaining samples are assigned to the noise set.
[0007] Preferably, in step S3, before the step of performing semantic consistency detection on the noisy images in the noise set using the pre-trained model, the following steps are included: Sort the samples from highest to lowest probability that they are noise samples; The highest probability samples are selected to form the noise set to be processed, and the label correction operation is performed only on the samples in this set.
[0008] Preferably, in step S3, the step of correcting the text labels corresponding to the noisy image when the preset enhancement conditions are met includes: Calculate the similarity between the noisy image and all text features in the candidate text library, and determine the optimal text that best matches the semantics of the noisy image; Calculate the similarity value between the optimal text and the noisy image, and subtract the similarity value between the original text and the noisy image to obtain the similarity difference; Determine whether the similarity difference is greater than a preset judgment threshold; If so, the original text of the noisy image is replaced with the optimal text; If not, then leave the original text unchanged.
[0009] Preferably, in step S4, the step of evaluating the impact of training samples on model performance using the knowledge base includes: Construct input data containing the current batch of training samples and their corresponding knowledge base entries; Calculate the loss value of the model on the knowledge base entries before and after training, respectively; Calculate the ratio of the loss value before training to the loss value after training; If the ratio is less than a preset value, it is determined that the training sample has a negative impact on the model performance, and the confidence weight of the training sample is reduced.
[0010] Preferably, in step S4, the step of generating confidence weights for each training sample based on the evaluation results includes: When it is determined that the training samples have a negative impact, the ratio is processed using the hyperbolic tangent function to generate a confidence weight less than one. When it is determined that the training samples have no negative impact, the confidence weight is set to a value of one.
[0011] Preferably, in step S5, the step of constructing the joint optimization objective that includes the pseudo-classification task includes: The text data in the clean set is considered as a category label; A classifier is trained using the data in the clean set to generate pseudo-classification prediction results for image data and pseudo-classification labels for text data. The cross-entropy loss between the pseudo-classification prediction result and the pseudo-classification label is calculated and used as the pseudo-classification loss term.
[0012] Preferably, in step S5, the step of constructing a joint optimization objective that includes a pseudo-classification task further includes: Calculate the entropy value of the pseudo-classification prediction result; An entropy regularization term is constructed to constrain the pseudo-classification prediction results to maintain a balanced distribution among the categories, preventing the classifier from mapping all samples to the same category.
[0013] Preferably, in step S5, the step of retraining the model using the confidence weights includes: The cross-entropy loss of the training samples is weighted using the confidence weights to obtain a weighted loss term; The model is computed to the guided loss term on the knowledge base. The weighted loss term, the guided loss term, the pseudo-classification loss term, and the entropy regularization term are added together to obtain the total loss function, and the model parameters are updated based on the total loss function.
[0014] Secondly, this invention provides the following technical solution: a robust cross-modal retrieval system based on noise label re-matching and pseudo-classification, the system comprising: The feature extraction module is used to acquire a cross-modal training set containing image and text data, and extract image features and text features respectively; The data preparation module is used to divide the cross-modal training set into a clean set and a noisy set based on the sample loss distribution characteristics, and to select samples from the clean set to build a knowledge base for evaluating model performance. The label correction module is used to perform semantic consistency detection on the noisy images in the noise set using a pre-trained model, and correct the text labels corresponding to the noisy images when the preset improvement conditions are met, so as to obtain the optimized sample correspondence. The weight generation module is used to evaluate the impact of training samples on model performance using the knowledge base, and generate confidence weights for each training sample based on the evaluation results. The model training module is used to construct a joint optimization objective that includes a pseudo-classification task, and to retrain the model using the confidence weights to obtain the final cross-modal retrieval model.
[0015] The present invention has the following beneficial effects: 1. In this invention, by introducing a noise label rematching mechanism, the semantic alignment capability of the pre-trained model is used to detect and correct erroneous image-text correspondences. At the same time, an adaptive similarity threshold is set to replace labels only when the semantic matching degree is significantly improved. This effectively utilizes the noise data that might otherwise be discarded, transforming it into effective training samples. It also avoids the risk of introducing additional noise due to blind replacement, thereby improving the annotation quality of cross-modal datasets and the final retrieval accuracy.
[0016] 2. In this invention, an impact-aware learning strategy based on a knowledge base is adopted. By constructing a clean sample set with high confidence as a knowledge base, the negative impact of training samples on model performance is evaluated in real time and confidence weights are dynamically generated accordingly. This can automatically identify and reduce the training weights of harmful samples, fundamentally blocking the misleading effect of noisy data on model parameter updates, preventing the neural network from overfitting and memorizing incorrect correspondences, and thus maintaining the stability of model performance in complex high-noise scenarios.
[0017] 3. In this invention, a pseudo-classification auxiliary task and entropy regularization constraint are constructed. The text title is innovatively regarded as a dynamic category label. The pseudo label is used to guide the model to perform classification training, which makes up for the deficiency of traditional contrastive learning in the lack of constraint in noisy environments. It promotes the formation of a more compact and clear cluster distribution in the feature space, effectively prevents model collapse, and enhances the discriminative and generalization capabilities of cross-modal feature representation. Attached Figure Description
[0018] Figure 1 This is a flowchart of the robust cross-modal retrieval method based on noise label re-matching and pseudo-classification proposed in this invention. Figure 2 This is a block diagram of the robust cross-modal retrieval method based on noise label rematching and pseudo-classification proposed in this invention. Detailed Implementation
[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] Example 1 In a first embodiment of the present invention, the present invention provides a robust cross-modal retrieval method based on noise label re-matching and pseudo-classification, such as... Figure 1 As shown, it includes the following steps: Step S1: Obtain a cross-modal training set containing image data and text data, and extract image features and text features respectively; Specifically, the first step is to construct or obtain a training dataset for the cross-modal retrieval task. This dataset is defined as follows: ,in Indicates the first One image-text sample pair, Represents image data, This represents the corresponding text description data. This represents the total number of sample pairs in the dataset. Since the data source is usually web crawling or crowdsourcing, there is an objectively certain proportion of incorrect matches in this original training set, meaning that some images... Its corresponding text They are not semantically consistent.
[0021] After acquiring the training data, a deep neural network model is used to extract features from the image and text data respectively. This embodiment adopts a two-stream network architecture, specifically including a visual encoder and a text encoder. As a specific implementation, a pre-trained CLIP model is used as the backbone network, where the visual encoder uses the ViT-B / 32 architecture and the text encoder uses a Transformer-based architecture.
[0022] For the visual modality, the input image The image is input to a visual encoder. The visual encoder segments the image into fixed-size patches and linearly maps these patches into sequence vectors. These vectors are then processed through multiple Transformer encoding layers, ultimately mapping the image into a high-dimensional feature space to obtain the image feature representation. This process can be represented as follows: ,in This represents the feature extraction function of the visual encoder. This is the extracted image feature vector.
[0023] For the text modality, the input text will be... The text is input to a text encoder. The text encoder first segments the text, adds start and end markers, and converts the tokens into word embedding vectors. These vectors are then processed through multiple Transformer encoding layers, ultimately mapping the text into the same common feature space as the image features, resulting in a text feature representation. This process can be represented as follows: ,in This represents the feature extraction function of the text encoder. This is the extracted text feature vector.
[0024] To measure the semantic similarity between images and text within the same common representation space, this embodiment extracts image feature vectors. and text feature vectors L2 normalization is performed separately. After normalization, the cosine similarity between the image features and text features is measured by calculating the dot product of the two features. For any image... and any text The similarity score between the two The calculation formula is as follows: ; In this formula, Representing an image Feature vector extracted by the visual encoder Representing text Feature vectors extracted by the text encoder; The L2 norm of a vector is used to normalize the magnitude of an eigenvector to 1. This represents the dot product operation of vectors.
[0025] Based on the above calculation method, correctly matched image-text pairs exhibit high similarity values in the common representation space, while unmatched sample pairs exhibit low similarity values. This feature extraction and similarity calculation process provides the basic data representation and metric for subsequent noise detection, label re-matching, and pseudo-classification training. In terms of specific hardware implementation, the above feature extraction process is executed on a computing device equipped with a graphics processing unit, utilizing a deep learning framework for computational acceleration.
[0026] Step S2: Based on the sample loss distribution characteristics, the cross-modal training set is divided into a clean set and a noisy set, and samples are selected from the clean set to build a knowledge base for evaluating model performance; In step S2, the step of dividing the cross-modal training set into a clean set and a noisy set based on the sample loss distribution characteristics includes: Calculate the cross-entropy loss value for each sample in the training set; The distribution of cross-entropy loss values is modeled using a Gaussian mixture model, and the posterior probability that each sample belongs to a clean sample is calculated. Samples with a posterior probability greater than a preset threshold are assigned to the clean set, and the remaining samples are assigned to the noise set. Specifically, firstly, for each image-text sample pair in the cross-modal training set... The symmetric cross-entropy loss value. Since cross-modal retrieval involves two directions—image-to-text retrieval and text-to-image retrieval—a bidirectional loss function is used to measure the degree of matching. For the first... For each sample pair, the symmetric cross-entropy loss The calculation formula is as follows: ; in, Represented by image The cross-entropy loss for anchor point retrieval text. Indicated by text The cross-entropy loss is used to retrieve the image for the anchor point. The specific calculation of the one-way cross-entropy loss is as follows: ; In the above formula, Indicates the first Modal samples With the Modal samples Cosine similarity between them; The size of the current batch; This is a temperature coefficient hyperparameter used to control the smoothness of the probability distribution. This calculation step yields the loss value for each sample in the training set. .
[0027] Secondly, based on the characteristic that deep neural networks tend to prioritize fitting simple, clean samples (with lower loss values) in the early stages of training and then fit noisy samples (with higher loss values) later, a Gaussian mixture model is used to analyze the loss distribution of all samples. Modeling is performed. This embodiment uses a Gaussian mixture model with two components, corresponding to the clean sample distribution and the noisy sample distribution, respectively. The probability density function of the loss value is... The definition is as follows: ; in, Indicates the number of Gaussian components; Indicates the first The mixing coefficients of Gaussian components, and satisfying ; Indicates the first The probability density function of Gaussian components, These represent the mean and variance of the component, respectively. The Gaussian mixture model parameters are iteratively estimated using the expectation-maximization algorithm, enabling the model to fit the current loss distribution.
[0028] After the model parameters converge, Bayes' theorem is used to calculate the posterior probability of each sample belonging to the "clean sample" component. The posterior probability Reflects the sample The confidence score for a correct match is calculated using the following formula: ; in, Indicates the index of the Gaussian component with the smaller mean. For the corresponding parameters.
[0029] Finally, based on the calculated posterior probability Divide the dataset. Set a probability threshold. All posterior probabilities The samples are divided into a clean set. , posterior probability The samples are divided into noise sets. The set is defined as follows: ； ; To build a knowledge base for evaluating model performance, from a clean set Samples with extremely high confidence are selected from the samples. In this embodiment, a strict threshold is set to ensure that the samples in the knowledge base contain almost no noisy correspondences. This knowledge base is used as benchmark data in subsequent training iterations to monitor and evaluate whether the model parameter updates are negatively affected by noisy samples, thereby providing a reliable basis for influencing the generation of perceptual weights.
[0030] Step S3: Use the pre-trained model to perform semantic consistency detection on the noisy images in the noise set. When the preset improvement conditions are met, correct the text labels corresponding to the noisy images to obtain the optimized sample correspondence. In step S3, prior to the step of performing semantic consistency detection on the noisy images in the noise set using the pre-trained model, the following steps are included: Sort the samples from highest to lowest probability that they are noise samples; The first part of the samples with the highest probability is selected to form the noise set to be processed, and the label correction operation is only performed on the samples in this set. In step S3, the step of correcting the text labels corresponding to the noisy image when the preset enhancement conditions are met includes: Calculate the similarity between the noisy image and all text features in the candidate text library, and determine the optimal text that best matches the semantics of the noisy image; Calculate the similarity value between the optimal text and the noisy image, and subtract the similarity value between the original text and the noisy image to obtain the similarity difference; Determine whether the similarity difference is greater than a preset threshold. If so, replace the original text in the noisy image with the optimal text; If not, then leave the original text unchanged; Specifically, in order to improve processing efficiency and concentrate computing resources to solve the high-confidence mislabeling problem, the noise set obtained in step S2 is first processed. Pre-screening is performed. The posterior probability is calculated based on the GMM model. The probability that a sample belongs to the noise category is defined as follows: . Set the noise All samples according to Sort the values from largest to smallest. Set a filtering ratio. The samples ranked highest after sorting are selected to form the subset to be corrected. For noisy samples that do not enter this subset, their original labels are kept unchanged, and they are not included in subsequent rematch calculations. This screening mechanism ensures that intervention is only applied to samples that are highly likely to be mislabeled, avoiding over-correction of ambiguous samples at the decision boundary.
[0031] For the subset to be corrected Each image sample in Therefore, it is necessary to find a semantically more suitable text tag for it. In this embodiment, a candidate text library is constructed. The text library contains all text data within the current batch, or comes from the entire clean set. The text data. Using the image features extracted in step S1. and the feature vectors of all texts in the candidate text library Calculate image With each candidate text Cosine similarity between .
[0032] Based on the calculated similarity distribution, images with similarity to the current noisy image are retrieved. The text with the highest similarity is denoted as the best candidate text. Its mathematical expression is: ; in, The operation is used to find the similarity function. The text parameter that yields the maximum value.
[0033] After determining the optimal candidate text Instead of directly replacing the text, a similarity-gain-based decision mechanism is introduced to prevent new noise from being introduced due to model prediction bias. First, the similarity value between the optimal candidate text and the image is calculated. and the original text corresponding to the image. Similarity value with image Then, the similarity difference between the two is calculated. : ; Set a preset judgment threshold. This threshold is used to measure the improvement in semantic consistency. The calculated similarity difference will be used... With the judgment threshold Comparison: like Then determine the original tag. There are obvious errors, and the optimal candidate text This can significantly improve semantic matching accuracy. At this point, a label correction operation is performed to adjust the sample... Updated to The updated samples are then added to the training process.
[0034] like If a slightly similar text was found, the improvement is insufficient to definitively prove the original label is incorrect or the new label is absolutely correct. To ensure data stability, no replacement operation is performed, and the original samples are preserved. constant.
[0035] Through the conditional label rematching process described above, the system can automatically correct samples with obvious semantic misalignments in noisy datasets, generating an optimized set of sample correspondences. This process significantly improves the label quality of training data without introducing manual intervention, providing more accurate supervision signals for subsequent model training.
[0036] Step S4: Use the knowledge base to evaluate the impact of training samples on model performance, and generate confidence weights for each training sample based on the evaluation results; In step S4, the step of evaluating the impact of training samples on model performance using the knowledge base includes: Construct input data containing the current batch of training samples and their corresponding knowledge base entries; Calculate the loss value of the model on knowledge base items before and after training; Calculate the ratio of the loss value before training to the loss value after training; If the ratio is less than the preset value, it is determined that the training sample has a negative impact on the model performance, and the confidence weight of the training sample is reduced. In step S4, the step of generating confidence weights for each training sample based on the evaluation results includes: When it is determined that the training samples have a negative impact, the ratio is processed using the hyperbolic tangent function to generate a confidence weight less than one. When it is determined that the training samples have no negative impact, the confidence weight is set to a value of one. Specifically, this step introduces the concept of influence functions from meta-learning, aiming to quantify the contribution of each training sample to the model's performance on a clean validation set. This process does not directly update the model parameters, but rather performs a virtual parameter update to probe the quality of the samples.
[0037] First, from the clean knowledge base built in step S2 A batch of validation data was randomly sampled and denoted as... Simultaneously, a batch of training samples in the current iteration step is obtained, denoted as... ; in For batch size, For image data, It is text data.
[0038] for Each training sample Perform the following evaluation process: Calculate the benchmark validation loss (pre-training loss): based on the parameter state of the current model. The computational model validates the data in the knowledge base. The average cross-entropy loss on the surface is denoted as... The calculation formula is as follows: ; in, This represents the validation set loss function. The cross-entropy loss function represents a single sample.
[0039] Perform virtual updates and compute new validation loss (post-training loss): using the current training samples For model parameters Perform a single-step virtual gradient descent update to obtain temporary parameters. This process is for evaluation only and does not change the parameters of the actual model. The update formula is: ; in, For virtual learning rate, This represents the gradient computation operator. Subsequently, based on the updated temporary parameters... The model is then recalculated using data validated on the same knowledge base. The loss value on is denoted as : ; Calculate the loss ratio and its impact on decision: Calculate the ratio of the baseline validation loss before training to the validation loss after virtual training. Used to measure samples Instantaneous impact on model generalization ability: ; Set a preset judgment threshold. .
[0040] like ,mean This means that after training with this sample, the model's loss on a clean knowledge base actually increased. This indicates that the sample may contain noisy or misleading information, negatively impacting model performance.
[0041] like ,mean If the validation set loss decreases or remains unchanged after training with the sample, it indicates that the sample is effective or at least harmless.
[0042] Generate confidence weights: Based on the above judgment results, generate the final confidence weights for each sample. When a sample is determined to have a negative impact, a hyperbolic tangent function is used to non-linearly map the loss ratio to generate a weight value less than 1, thereby suppressing the contribution of that sample in subsequent training. The calculation formula is as follows: ; in, For adjustment coefficients, The range of the function is (-1, 1). Since the loss values are all positive, ,therefore This achieves a reduction in the weighting of harmful samples. When a sample is determined not to have a negative impact, the confidence weight is directly set to the maximum value. ; Through this step, the system can dynamically assign a weight to each training sample that reflects its "learning value". This weight will then be passed to the joint optimization objective in step S5 to adjust the model's attention to different samples.
[0043] Step S5: Construct a joint optimization objective that includes the pseudo-classification task, and retrain the model by combining the confidence weights to obtain the final cross-modal retrieval model; In step S5, the steps for constructing the joint optimization objective that includes the pseudo-classification task include: Treat the text data in the clean set as category labels; A classifier is trained using data from a clean set to generate pseudo-classification predictions for image data and pseudo-classification labels for text data. Calculate the cross-entropy loss between the pseudo-classification prediction result and the pseudo-classification label, and use it as the pseudo-classification loss term; In step S5, the step of constructing the joint optimization objective that includes the pseudo-classification task further includes: Calculate the entropy value of the pseudo-classification prediction result; Construct an entropy regularization term to constrain the pseudo-classification prediction results to maintain a balanced distribution among the categories, preventing the classifier from mapping all samples to the same category; In step S5, the steps of retraining the model using confidence weights include: The cross-entropy loss of the training samples is weighted using confidence weights to obtain a weighted loss term; The guided loss term of the computational model on the knowledge base; The weighted loss term, the guided loss term, the pseudo-classification loss term, and the entropy regularization term are added together to obtain the total loss function, and the model parameters are updated based on the total loss function. Specifically, to further explore the latent semantic structure of the data and prevent model degradation under noisy labels, this embodiment constructs an auxiliary pseudo-classification task. Unlike traditional instance-level contrastive learning, this task uses the clean set selected in step S2... The text data in the training batch is treated as fixed classification label categories. For each image sample in the current training batch... Input it into the model to extract features And calculate the image features and all of them in the clean set. Text features The similarity is used. Based on this similarity, the Softmax function is used to generate the pseudo-classification prediction probability distribution of the image belonging to each text category. For the first Predicted probabilities for each category The calculation is as follows: ; in, This is the temperature coefficient for the pseudo-classification task. Simultaneously, corresponding pseudo-classification labels are generated. If the image If a text originally belonged to the clean set, its corresponding text index is the correct label; if it belonged to the noisy set, the text index after rematching in step S3 is used as the label. The pseudo-classification loss term is calculated accordingly. : ; in, For batch size, This is an indicator function.
[0044] To prevent model collapse during pseudo-classification, where the classifier predicts all images as a few specific text categories, this embodiment introduces an entropy regularization term, Lreg. This regularization term aims to constrain the average prediction distribution Pˉ of all images within the current batch to remain balanced. First, the average prediction probability distribution within the batch is calculated. To make this distribution as close as possible to a uniform distribution (i.e., to maximize its information entropy), the following regularized loss function is constructed: ; By minimizing this term (i.e. maximizing) This forces the model to maintain diversity in class predictions and avoid getting trapped in local optima.
[0045] During the model parameter update phase, the confidence weights generated in step S4 are used. The basic cross-modal retrieval loss is weighted to obtain the weighted loss term. This ensures that high-confidence samples dominate the gradient descent process, while the contribution of low-confidence noisy samples is suppressed. The calculation formula is: ; In addition, to ensure that the model does not deviate from the correct data distribution during the learning process, a guiding loss term is calculated using clean samples from the knowledge base. This involves sampling a batch of data from the knowledge base, calculating its standard cross-entropy loss, and using it as an anchor point for model optimization.
[0046] Finally, the weighted sum of all the above loss terms yields the total loss function used for backpropagation. : ; in, To balance the hyperparameters of the weights of each loss term, based on this total loss function... By using stochastic gradient descent or the Adam optimizer to iteratively update the parameters of the deep neural network until the model converges, a highly robust cross-modal retrieval model is finally obtained.
[0047] Example 2: When constructing cross-modal retrieval systems for social media or internet content, the training data, widely derived from user uploads or web crawling, commonly exhibits a "noise association" phenomenon where the semantics of textual descriptions and visual content are inconsistent. To address this issue, this invention employs a robust cross-modal retrieval system based on noise label re-matching and pseudo-classification, the structure of which is as follows: Figure 2 As shown. The specific implementation process of this system is as follows: The feature extraction module is used to acquire a cross-modal training set containing image and text data, and extract image features and text features respectively; The data preparation module is used to divide the cross-modal training set into a clean set and a noisy set based on the sample loss distribution characteristics, and to select samples from the clean set to build a knowledge base for evaluating model performance. The label correction module is used to perform semantic consistency detection on noisy images in the noise set using a pre-trained model. When the preset improvement conditions are met, the text labels corresponding to the noisy images are corrected to obtain the optimized sample correspondence. The weight generation module is used to evaluate the impact of training samples on model performance using a knowledge base, and generate confidence weights for each training sample based on the evaluation results. The model training module is used to construct a joint optimization objective that includes a pseudo-classification task, and to retrain the model by combining confidence weights to obtain the final cross-modal retrieval model.
[0048] Specifically, in the feature extraction stage, the system receives raw image-text pairs from internet crawling or user uploads. For image data, the system uses a deep convolutional neural network as an image encoder to map the input image into a high-dimensional visual feature vector. This process captures the image's texture, shape, and semantic information. For text data, the system uses a pre-trained language model based on the Transformer architecture as a text encoder to map the corresponding text description into a high-dimensional text feature vector. The two encoders share the same embedding space dimension, ensuring that visual features and text features have the mathematical basis for calculating dot product or cosine similarity.
[0049] During the data preparation phase, the system calculates the cross-entropy loss value for each pair of samples on the pre-trained model. Based on the memory effect principle that "small loss samples correspond to high confidence labels," the system classifies samples with loss values less than a preset threshold as a clean set and samples with loss values greater than the preset threshold as a noisy set. The system further extracts a portion of high-confidence samples from the clean set based on the principles of representativeness and balance of feature distribution to construct a metadata knowledge base. This knowledge base serves as the "gold standard" for guiding subsequent model updates, providing clean supervisory signals and addressing the lack of authoritative benchmarks in social media data.
[0050] During the label correction phase, for "dirty data" in the noise set that is judged to have a semantic mismatch between the image and text, the system introduces a large-scale vision-language model pre-trained on a large-scale general dataset as an auxiliary detector. The system calculates the semantic consistency score between the noisy image and the current text label, and simultaneously retrieves the Top-K text candidates that are closest to the visual features of the image in the text feature library. When the consistency score of the original label is lower than that of the candidate text and the difference exceeds a preset tolerance, the system performs a label re-matching operation, replacing the original noisy text with the target text with the highest semantic matching degree among the candidate texts, thereby generating semantically calibrated optimized sample pairs.
[0051] During the weight generation phase, the system introduces a meta-learning-based weight evaluation mechanism. The system inputs samples from the current training batch into the model, calculates their gradient directions under the current parameters, and applies the updated parameters to the validation samples in the knowledge base. The system monitors the change in validation set loss of knowledge base samples after parameter updates: if a training sample causes a decrease in validation set loss, the system determines that the sample has a positive contribution to model performance and assigns it a high-confidence weight; if it causes an increase in validation set loss, the system determines that the sample is harmful noise and assigns it a low weight or zero weight. This process generates a dynamic scalar weight for each sample that reflects its authenticity.
[0052] During the model training phase, the system constructs a joint optimization objective that includes cross-modal retrieval and unsupervised pseudo-classification tasks. In the cross-modal retrieval task, the system combines the previously generated confidence weights to minimize the weighted image-text matching loss, thereby reducing the interference of residual noise on feature alignment. Simultaneously, in the pseudo-classification task, the system performs clustering operations on the visual and textual features of the full dataset, generating pseudo-labels that reflect the underlying semantic structure of the data, and forces the model to predict these pseudo-labels. Through joint optimization, the system corrects noisy labels while mining the intrinsic category information of the data, ultimately outputting a robust cross-modal retrieval model that provides accurate visual content feedback for complex user query intents.
[0053] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A robust cross-modal retrieval method based on noise label re-matching and pseudo-classification, characterized in that, include: Step S1: Obtain a cross-modal training set containing image data and text data, and extract image features and text features respectively; Step S2: Based on the sample loss distribution characteristics, the cross-modal training set is divided into a clean set and a noisy set, and samples are selected from the clean set to construct a knowledge base for evaluating model performance; Step S3: Use a pre-trained model to perform semantic consistency detection on the noisy images in the noise set. When the preset improvement conditions are met, correct the text labels corresponding to the noisy images to obtain the optimized sample correspondence. Step S4: Use the knowledge base to evaluate the impact of training samples on model performance, and generate confidence weights for each training sample based on the evaluation results; Step S5: Construct a joint optimization objective that includes the pseudo-classification task, and retrain the model using the confidence weights to obtain the final cross-modal retrieval model.
2. The robust cross-modal retrieval method based on noise label re-matching and pseudo-classification according to claim 1, characterized in that, In step S2, the step of dividing the cross-modal training set into a clean set and a noisy set based on the sample loss distribution characteristics includes: Calculate the cross-entropy loss value for each sample in the training set; The distribution of the cross-entropy loss value is modeled using a Gaussian mixture model, and the posterior probability that each sample belongs to a clean sample is calculated. Samples with a posterior probability greater than a preset threshold are assigned to the clean set, and the remaining samples are assigned to the noise set.
3. The robust cross-modal retrieval method based on noise label re-matching and pseudo-classification according to claim 1, characterized in that, In step S3, prior to the step of performing semantic consistency detection on the noisy images in the noise set using a pre-trained model, the following steps are included: Sort the samples from highest to lowest probability that they are noise samples; The highest probability samples are selected to form the noise set to be processed, and the label correction operation is performed only on the samples in this set.
4. The robust cross-modal retrieval method based on noise label re-matching and pseudo-classification according to claim 1, characterized in that, In step S3, the step of correcting the text labels corresponding to the noisy image when the preset enhancement conditions are met includes: Calculate the similarity between the noisy image and all text features in the candidate text library, and determine the optimal text that best matches the semantics of the noisy image; Calculate the similarity value between the optimal text and the noisy image, and subtract the similarity value between the original text and the noisy image to obtain the similarity difference; Determine whether the similarity difference is greater than a preset judgment threshold; If so, the original text of the noisy image is replaced with the optimal text; If not, then leave the original text unchanged.
5. The robust cross-modal retrieval method based on noise label re-matching and pseudo-classification according to claim 1, characterized in that, In step S4, the step of evaluating the impact of training samples on model performance using the knowledge base includes: Construct input data containing the current batch of training samples and their corresponding knowledge base entries; Calculate the loss value of the model on the knowledge base entries before and after training, respectively; Calculate the ratio of the loss value before training to the loss value after training; If the ratio is less than a preset value, it is determined that the training sample has a negative impact on the model performance, and the confidence weight of the training sample is reduced.
6. The robust cross-modal retrieval method based on noise label re-matching and pseudo-classification according to claim 1, characterized in that, In step S4, the step of generating confidence weights for each training sample based on the evaluation results includes: When it is determined that the training samples have a negative impact, the ratio is processed using the hyperbolic tangent function to generate a confidence weight less than one. When it is determined that the training samples have no negative impact, the confidence weight is set to a value of one.
7. The robust cross-modal retrieval method based on noise label re-matching and pseudo-classification according to claim 1, characterized in that, In step S5, the step of constructing a joint optimization objective that includes the pseudo-classification task includes: The text data in the clean set is considered as a category label; A classifier is trained using the data in the clean set to generate pseudo-classification prediction results for image data and pseudo-classification labels for text data. The cross-entropy loss between the pseudo-classification prediction result and the pseudo-classification label is calculated and used as the pseudo-classification loss term.
8. The robust cross-modal retrieval method based on noise label re-matching and pseudo-classification according to claim 1, characterized in that, In step S5, the step of constructing a joint optimization objective that includes a pseudo-classification task further includes: Calculate the entropy value of the pseudo-classification prediction result; An entropy regularization term is constructed to constrain the pseudo-classification prediction results to maintain a balanced distribution among the categories, preventing the classifier from mapping all samples to the same category.
9. The robust cross-modal retrieval method based on noise label re-matching and pseudo-classification according to claim 1, characterized in that, In step S5, the step of retraining the model using the confidence weights includes: The cross-entropy loss of the training samples is weighted using the confidence weights to obtain a weighted loss term; The model is computed to the guided loss term on the knowledge base. The weighted loss term, the guided loss term, the pseudo-classification loss term, and the entropy regularization term are added together to obtain the total loss function, and the model parameters are updated based on the total loss function.
10. A robust cross-modal retrieval system based on noise label re-matching and pseudo-classification, characterized in that, The system for the robust cross-modal retrieval method based on noise label re-matching and pseudo-classification as described in any one of claims 1-9 comprises: The feature extraction module is used to acquire a cross-modal training set containing image and text data, and extract image features and text features respectively; The data preparation module is used to divide the cross-modal training set into a clean set and a noisy set based on the sample loss distribution characteristics, and to select samples from the clean set to build a knowledge base for evaluating model performance. The label correction module is used to perform semantic consistency detection on the noisy images in the noise set using a pre-trained model, and correct the text labels corresponding to the noisy images when the preset improvement conditions are met, so as to obtain the optimized sample correspondence. The weight generation module is used to evaluate the impact of training samples on model performance using the knowledge base, and generate confidence weights for each training sample based on the evaluation results. The model training module is used to construct a joint optimization objective that includes a pseudo-classification task, and to retrain the model using the confidence weights to obtain the final cross-modal retrieval model.