Noise label cross-modal retrieval method based on gradient guided dynamic division

Abstract

The invention provides a noise label cross-modal retrieval method based on gradient guide dynamic division, and belongs to the technical field of computers. According to the cross-modal retrieval method with the noise label, the problem that the noise label in multi-modal data affects the cross-modal retrieval precision can be solved, and the noise label sample is identified and processed by constructing a dynamic division strategy based on a sample loss function gradient. The method comprises the steps of common semantic representation learning, gradient-guided screening and correction, dynamic division and robust learning, cross-modal consistency learning and overall objective function construction, and finally cross-modal cross retrieval is carried out through the optimized overall objective function. According to the method, the multi-modal data with noise can be effectively utilized, and the accuracy and robustness of cross-modal retrieval are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and relates to a cross-modal retrieval method with noisy labels for various modal data in the Internet, and in particular to a cross-modal retrieval method with noisy labels based on gradient-guided dynamic partitioning. Background Art

[0002] With the popularization of digital technology and the widespread application of the Internet, Internet multimedia data (such as pictures, videos, and audio shared by users) has shown explosive growth, and cross-modal retrieval plays a vital role in the field of multimedia information retrieval. Cross-modal retrieval refers to using data of one modality as a query to retrieve data of another modality related to it. In practical applications, it includes image query text, text-based audio retrieval, video recommendation, visual question answering and other tasks. However, the effect of cross-modal retrieval usually depends on the high-quality annotation of multimodal data. Most existing methods assume that data labels are complete and accurate, and that data between different modalities are completely paired. For example, Jiangnan University proposed a cross-modal retrieval method based on graph convolution (Chinese invention patent CN202110760740.5), which discloses a cross-modal retrieval technology based on graph convolution. And Huaqiao University and Xiamen Yealink Network Technology Co., Ltd. jointly proposed a cross-modal retrieval method based on neighbor ranking relationship (Chinese invention patent CN202111564321.0), which discloses a more refined cross-modal retrieval technology based on neighbor ranking relationship. Both methods require that the data used to train their models is high-quality and well-annotated. However, collecting large-scale, well-annotated multimodal datasets is a time-consuming and expensive process. To reduce costs, researchers have proposed a variety of methods, such as web scraping, crowdsourcing, and automatic annotation using pre-trained models. However, these methods inevitably introduce noisy labels. Since Deep Neural Networks (DNNs) are prone to overfitting to noisy labels during training, noisy samples may be incorrectly assigned to a given label category, thereby reducing the generalization ability of the model. In an environment with noisy labeled data, achieving cross-modal retrieval is a challenging task, and this issue has not received enough attention so far.

[0003] Therefore, it is of great research significance to propose an effective method to improve the robustness and generalization ability of DNN in cross-modal retrieval under noisy label environment. Although many unsupervised methods have tried to avoid the interference of noisy labels, their performance is usually significantly lower than that of supervised methods. Noisy labels may confuse the discriminative associations between cross-modal data, making it difficult to eliminate the differences between heterogeneous modalities, thereby reducing the performance of cross-modal retrieval. When faced with multimodal data with noisy labels, traditional cross-modal retrieval methods are prone to overfitting the noisy label data and are difficult to effectively deal with the noisy label problem.

[0004] In this regard, researchers have proposed two main methods: one is to design a robust loss function, and the other is to screen and correct noisy labels. These methods aim to enhance the adaptability and robustness of the model to noisy labels. However, these methods still have limitations. Although designing a robust loss function or regularization term can enhance the model's ability to resist noise, the model may eventually overfit these noisy data due to the presence of noisy labels in the training data, and the generalization ability of the model decreases significantly as the proportion of noise increases. Other methods attempt to distinguish clean data from noisy data in the training data, so as to use clean data to transform it into a purely supervised cross-modal retrieval problem, or to make full use of the valuable information in the noisy data through label correction. However, these methods are usually limited by the screening and correction of noisy labels, and it is difficult to accurately correct noisy samples. Therefore, studying how to effectively distinguish noisy data from clean data and appropriately correct noisy labels to improve data quality is the key to solving this problem. Summary of the invention

[0005] In view of the above problems existing in the prior art, in order to solve the problem of cross-retrieval of multimodal data in the Internet, the present invention provides a cross-modal retrieval method with noisy labels based on gradient-guided dynamic partitioning, which is a cross-modal retrieval solution for dealing with noisy labels and can effectively improve the robustness and accuracy of retrieval.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A cross-modal retrieval method with noisy labels based on gradient-guided dynamic partitioning, wherein the gradient guidance in the cross-modal retrieval method with noisy labels is to partition the data set and correct the noisy labels through gradient vectors. The dynamic partitioning is to divide the data set into three subsets: clean, noisy, and difficult during the model training process. The cross-modal retrieval method with noisy labels reduces the heterogeneity and semantic gap between different modalities under the influence of noisy labels. It includes the following steps:

[0008] Step 1: Common semantic representation learning, specifically: project each modality into the common semantic space and calculate the class prototype probability of the sample to achieve correct classification of the sample.

[0009] Step 2, gradient-guided screening and correction, specifically: for the data after learning the common semantic representation, use the gradient information to screen out the clean label data and correct some noisy label data, thereby improving the quality of the dataset.

[0010] Step 3, dynamic partitioning and robust learning, specifically: dynamically partition the dataset screened and corrected in step 2 into three subsets: clean, noisy, and difficult, and apply different robust loss functions on each subset to mitigate the impact of noisy labels.

[0011] Step 4, cross-modal consistency learning, specifically: based on the dynamic partitioning and robust learning in step 3, a contrastive learning strategy is adopted to ensure that the representations of samples of different modalities of the same target in the common feature space are close to each other, while samples of different targets should be far away.

[0012] Step 5: Construct an overall objective function and optimize the model performance by minimizing the overall objective function.

[0013] Furthermore, the step 1 is specifically as follows:

[0014] Step 1.1, let represents a multimodal dataset with m modalities, represents the i-th modality (such as image or video modality), each modality contains N samples. represents the jth sample of the i-th mode, Representation sample The label of, where K represents the number of semantic categories, the corresponding one-hot vector is expressed as .

[0015] In view of the heterogeneous differences between different modalities, the present invention proposes to use the pre-trained CLIP (Contrastive Language-Image Pre-Training) as the backbone network, learn the modality-specific transformation function through the features extracted by the backbone network, and project the image or video modality into a common semantic space. Therefore, the process of extracting features from each sample through the backbone network is shown in formula (1):

[0016] (1);

[0017] in, Represent the transformation functions of different modes; Represents the multimodal features (graphic and text features) extracted by CLIP.

[0018] Step 1.2, after obtaining the multimodal features extracted by CLIP, stack a three-layer fully connected network to project the extracted multimodal features into a common semantic space; as shown in formula (2):

[0019] (2);

[0020] in, represents the feature extraction network of the i-th modality; represents the learnable parameter matrix; represents the dimension of the public semantic space, It means that the dimension of the result after the transformation of the formula is ; Represents the embedding representation of each sample in the common semantic space.

[0021] Step 1.3, in the public semantic space, calculate the class prototype probability distribution of the sample to achieve the correct classification of the sample. For each category, calculate the average value of all its samples in the public semantic space, and use it as the class prototype of the category. Construct the class prototype as the anchor point of clustering ,in Represents the prototypes of k categories (the same as the number of semantic categories mentioned above). Calculate the similarity between each sample and each class prototype to achieve discriminative analysis of sample features, as shown in formula (3):

[0022] (3);

[0023] in, Represents the similarity vector between the sample and each class prototype; Represents the matrix of all class prototype vectors connected in parallel and then transposed.

[0024] Step 1.4, Sample The probability of belonging to each class is shown in formula (4):

[0025] (4);

[0026] in, represents the temperature parameter, which is used to adjust the smoothness of the output probability; Representation sample The probability vector belonging to each class, each bit in the vector is the predicted probability value of the sample belonging to a certain category.

[0027] By minimizing The standard loss between the given labels is used to optimize learning, such as the cross-entropy loss (CE). Since the data collected by the multimodal dataset has noisy labels, the loss function cannot be simply applied directly to the multimodal retrieval training with noisy labels, but needs to be processed in subsequent steps.

[0028] Furthermore, the step 2 is specifically as follows:

[0029] In order to deal with the problem of multimodal retrieval with noisy labels, the present invention proposes a gradient-guided screening and correction strategy to screen out reliable clean label data and correct some noisy label data. In the model optimization process, the loss function is used to quantify the difference between the predicted output of the model and the given label, and its main goal is usually to minimize the loss function, thereby reducing the difference between the sample and its corresponding label. The gradient is a vector that indicates the direction in which the function grows fastest, and its magnitude indicates the rate of change of the function. The process of minimizing the loss function includes optimizing in the opposite direction of the gradient to obtain the minimum value of the loss function. During the minimization of the loss function, the predicted distribution of the sample is optimized to be as closely aligned as possible with the one-hot vector distribution of the given label. The model expects that the predicted value of the sample category relative to the given label is close to 1, while the predicted values ​​of other categories should be close to 0. In order to minimize the loss, the gradient calculation of this method takes into account the gap between the predicted distribution of the sample and the given label. Therefore, in the optimization process, more attention is paid to and highlights the samples with large differences between the predicted values ​​and the given labels, so as to quickly adjust the model parameters and achieve the purpose of convergence. Specifically, the following steps are included:

[0030] Step 2.1, use the cross entropy loss to preliminarily calculate the gradient value distribution of all samples, and calculate the cross entropy loss function of the predicted probability value obtained by formula (4) and the one-hot vector of the given label, as shown in formula (5):

[0031] (5);

[0032] in, represents the unique hot vector corresponding to each sample label, m represents the number of modes, N represents the number of samples, Represents the value of the cross entropy loss function.

[0033] Step 2.2, for the cross entropy loss function The gradient calculation of is shown in formula (6):

[0034] (6);

[0035] Step 2.3, for the sample The gradient is reflected in its similarity to the prototype of each class According to the given label gradient calculation, there are two cases, as shown in formula (7):

[0036] (7);

[0037] Among them, cls represents the current sample label category. When the sample prediction label is the same as the true label, the corresponding gradient decreases by one.

[0038] Step 2.4, calculate the L2 norm of the gradient as the screening confidence of the noisy sample, as shown in formula (8):

[0039] (8);

[0040] in, Represents L2 norm, filter confidence The smaller ones are clean samples, and the rest are noise samples.

[0041] Step 2.5, in addition, in order to improve the availability of noise samples, some of the noise samples are corrected. In order to avoid introducing more noise during correction, the correction confidence of each sample is calculated, as shown in formula (9):

[0042] (9);

[0043] in, It means to find the maximum value of sample gradient. It means to find the minimum value of sample gradient. The higher the correction confidence, the greater the possibility that the sample is corrected.

[0044] When the correction confidence meets the specified threshold condition, the noisy labeled sample will be assigned a pseudo label, that is, the category with the largest gradient value, as shown in formula (10):

[0045] (10);

[0046] in, Indicates the category index corresponding to the maximum gradient value; The value representing the pseudo label.

[0047] Furthermore, the step 3 is specifically as follows:

[0048] In order to prevent the model from overfitting to data with noisy labels, the present invention further proposes a dynamic partitioning and robust learning strategy based on screening confidence and correction confidence. The multimodal dataset is divided into three subsets: a clean subset, a noisy subset, and a difficult subset. Since the noise labels still exist in each subset with different proportions, different robust loss functions are used on each subset to reduce the impact of noise. The following steps are included:

[0049] Step 3.1, first, the selection and learning of the clean subset. According to the analysis of step 2.4, the smaller the L2 norm of the gradient, the higher the probability that the sample is clean. Therefore, the present invention selects the clean subset as:

[0050] (11);

[0051] in, represents small batches of data, Represents the i-th modal data; e i = sort(G( M ˜ i ))[| M ˜ i |*R(t)] represents the screening threshold of the i-th mode, represents the percentage of clean samples selected from the small batch; represents the function described in formula (8), Represents a sorting function.

[0052] In addition, in order to ensure that most samples remain in the clean subset to dominate model training before overfitting, the present invention proposes a dynamic selection strategy. It is initially set to a large value, and then gradually decreases with the training rounds until it stabilizes. Since the clean subset C has a more accurate label, the cross entropy loss is used for optimization of the clean subset, as shown in formula (12):

[0053] (12);

[0054] in, Represents the size of the clean subset C; Represents the loss value obtained by the cross entropy loss function for the data in the clean subset.

[0055] Step 3.2, followed by the selection and learning of noise subsets, in order to utilize the noise data outside the clean subset, the samples whose correction confidence is greater than the correction threshold are grouped into noise subsets , as shown below:

[0056] (13);

[0057] in, Indicates the correction threshold, the value range is between 0-1; Indicates samples whose correction confidence is higher than the correction threshold; represents the pseudo label corrected according to formula (10); Represents mini-batches of data.

[0058] Since the corrected pseudo-labels may still be inaccurate, the model will be disturbed by inaccurate labels if it is directly optimized using cross entropy loss. Therefore, a more robust loss is adopted, namely generalized cross entropy loss, which combines the advantages of mean absolute error (MAE) and cross entropy (CE) and is defined on the noisy subset. The generalized cross entropy loss function is shown in formula (14):

[0059] (14);

[0060] in, represents the noise subset; r ∈ （ 0 ， 1 ] , represents the adjustment parameter, which is used to balance robustness and learning efficiency. Close to 0, use L'Oreal ( ) law, the generalized cross entropy loss is equivalent to the cross entropy, and as it approaches 1, it will approach the mean absolute error.

[0061] Step 3.3, finally, the selection and learning of the difficult subset. After the above screening, the clean and noisy samples are excluded, and the remaining samples in the multimodal dataset are regarded as difficult samples. .

[0062] The samples in the difficult subset are ambiguous and difficult to correct. In order to avoid adding external noise, the present invention proposes to learn the common distribution of prediction probabilities from different modalities of paired samples, that is, distribution consistency learning. Therefore, for the difficult subset, a distribution consistency learning strategy is proposed, which uses the mean squared error (MSE) loss function to align the prediction distributions between different modalities of samples.

[0063] Since this property is applicable to all samples, it is extended to the entire multimodal dataset as shown in formula (15):

[0064] (15);

[0065] in, Indicates the batch size; Represents the predicted probability distribution of other modal data except the current i-th modality, and stipulates ≠ ; Represents the value of the mean squared error loss function.

[0066] Furthermore, the step 4 is specifically as follows:

[0067] In order to ensure that the embeddings of paired samples from different modalities are well aligned in the common feature space, in addition to the prediction label probability distribution consistency learning in step 3, the present invention also proposes a cross-modal consistency learning strategy. Through contrastive learning, the common semantic representations of different modalities of the same target are used as positive sample pairs. The distribution of positive samples should be close to each other, while other samples as negative samples should be far away. Therefore, the multimodal contrastive learning loss function is shown in formula (16):

[0068] (16);

[0069] in, is the temperature parameter; Indicates The embedding representation of the jth sample in the modality in the common semantic space.

[0070] Furthermore, in step 5, an overall objective function is constructed, and the overall objective function is formed by combining formula (12), formula (14), formula (15) and formula (16), and the performance of the model is optimized by minimizing the overall objective function, specifically:

[0071] In order to make the model converge gradually, in the initial stage, a training warm-up process is used, namely, cross entropy loss to realize the calculation of screening confidence and correction confidence, as shown in formula (5). Through warm-up, the screening and correction of noise samples are realized. Combining three different types of robust loss functions, the overall objective function of the present invention is shown in formula (17):

[0072] (17);

[0073] in, It is a balance parameter that adjusts the weights of the three loss functions.

[0074] By minimizing the overall objective function, the stochastic gradient descent optimization algorithm is used for iterative optimization to obtain the optimized overall objective function, and then cross-modal cross retrieval is performed.

[0075] Innovation analysis of the present invention: (1) The present invention can process multi-modal data such as images, texts and videos, and realize a wide range of cross-modal retrieval applications. (2) The present invention proposes a gradient-guided dynamic partitioning method for cross-modal retrieval with noisy labels, solving a widespread but rarely explored problem. (3) The present invention uses the gradient information of samples to screen clean samples and correct noisy labels, thereby improving the quality of multi-modal datasets. (4) The present invention enhances the robustness and discrimination ability of the model by dynamically partitioning the dataset and using specific loss functions for different subsets.

[0076] The beneficial effects of the present invention are:

[0077] (1) Dynamic partitioning strategy. A dynamic partitioning strategy based on gradient guidance is proposed, which can effectively distinguish clean samples from noisy samples by using the gradient information of samples before the model overfits. The dataset is partitioned by gradient vectors and the noise labels are corrected, and the dataset is further dynamically divided into clean, noisy, and difficult subsets. This strategy not only improves the efficiency of data utilization, but also enhances the model's adaptability to different types of data.

[0078] (2) Application of robust loss function. Specific robust loss functions are designed for data subsets of different qualities, and clean samples, noisy samples, and difficult samples are learned separately. This differentiated processing strategy can effectively improve the model's discriminative ability and robustness in complex data environments.

[0079] (3) Cross-modal consistency strategy. A cross-modal consistency strategy is adopted to reduce the heterogeneous differences between modalities and realize multimodal knowledge retrieval with noisy labels. This strategy promotes the consistency of data distribution between modalities through multimodal contrast learning loss function, reduces modal heterogeneity and semantic gaps in the common semantic space. This innovation significantly improves the robustness of cross-modal retrieval and provides a more reliable solution for multimodal learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 Flowchart of the cross-modal retrieval method with noisy labels. DETAILED DESCRIPTION

[0081] The implementation of the present invention is further described below in conjunction with cross-searching between images and texts on the Internet.

[0082] A cross-modal retrieval method with noisy labels based on gradient-guided dynamic partitioning is specifically implemented in the following steps:

[0083] The step 1, common semantic representation learning, projects each modality into the common semantic space and calculates the class prototype probability of the sample to achieve correct classification of the sample, specifically:

[0084] Step 1.1, let represents a multimodal dataset with m modalities, represents the i-th modality (such as image or video modality), each modality contains N samples. represents the jth sample of the i-th mode, Representation sample The label of, where K represents the number of semantic categories, the corresponding one-hot vector is expressed as This embodiment selects three data sets to conduct experiments, namely, Wikipedia, Pascal-Sentence, and XMediaNet data sets.

[0085] The Wikipedia dataset includes 2866 image-text pairs from WikiPedia, each with ten annotated ground truth concepts. Pascal-Sentence includes 1000 pairs of images and corresponding text descriptions extracted from VOC2008, covering ten semantic categories. XMediaNet is a comprehensive multimodal dataset containing five modalities and covering 200 different categories. Table 1 shows the detailed statistics of each dataset.

[0086] Table 1. Statistics of the datasets used in the application examples.

[0087]

[0088] In view of the heterogeneous differences between different modalities, the present invention proposes to use the pre-trained CLIP (Contrastive Language-Image Pre-Training) as the backbone network, learn the modality-specific transformation function through the features extracted by the backbone network, and project the image or video modality into a common semantic space. Therefore, the process of extracting features from each sample through the backbone network is shown in formula (1):

[0089] (1);

[0090] in, Represent the transformation functions of different modes; Represents the features of the image and text extracted by CLIP.

[0091] Step 1.2, after obtaining the multimodal features extracted by CLIP, stack a three-layer fully connected network to project the extracted multimodal features into a common semantic space; as shown in formula (2):

[0092] (2);

[0093] in, represents the feature extraction network of the i-th modality, represents the learnable parameter matrix, Indicates the dimension of the public semantic space. In the experiment, the value of l is 512. It means that the dimension of the result after the transformation of the formula is , Represents the embedding representation of each sample in the common semantic space.

[0094] Step 1.3, in the common semantic space, calculate the class prototype probability distribution of the sample to achieve the correct classification of the sample. For each category, this method calculates the average value of all its samples in the semantic space as the class prototype of the category, and constructs the class prototype as the anchor point of clustering ,in Represents the prototypes of k categories. The similarity between each sample and each class prototype is calculated to achieve discriminative analysis of sample features, as shown in formula (3):

[0095] (3);

[0096] in, Represents the similarity vector between the sample and each class prototype; Represents the matrix of all class prototype vectors connected in parallel and then transposed.

[0097] Step 1.4, Sample The probability of belonging to each class is shown in formula (4):

[0098] (4);

[0099] in, represents the temperature parameter, which is used to adjust the smoothness of the output probability; Represents the probability vector of the sample belonging to each class. By minimizing Standard loss between given labels to optimize learning, such as CrossEntropy Loss (CE);

[0100] Since the data collected by the multimodal dataset has noisy labels, the loss function cannot be simply applied directly to the multimodal retrieval training with noisy labels, but needs to be processed in subsequent steps.

[0101] Furthermore, the step 2, gradient-guided screening and correction, uses gradient information to screen out clean label data and correct some noisy label data for the data after learning the common semantic representation, thereby improving the quality of the data set, specifically:

[0102] Step 2.1, use the cross entropy loss to preliminarily calculate the gradient value distribution of all samples, and calculate the cross entropy loss function of the predicted probability value obtained by formula (4) and the one-hot vector of the given label, as shown in formula (5):

[0103] (5);

[0104] in, represents the unique hot vector corresponding to each sample label, m represents the number of modes, N represents the number of samples, Represents the value of the cross entropy loss function.

[0105] Step 2.2, for the cross entropy loss function The gradient calculation of is shown in formula (6):

[0106] (6);

[0107] Step 2.3, for the sample The gradient is reflected in its similarity to the prototype of each class According to the given label gradient calculation, there are two cases, as shown in formula (7):

[0108] (7);

[0109] Among them, cls represents the current sample label category. When the sample prediction label is the same as the true label, the corresponding gradient decreases by one.

[0110] Step 2.4, calculate the L2 norm of the gradient as the screening confidence of the noisy sample, as shown in formula (8):

[0111] (8);

[0112] in, Represents L2 norm, filter confidence The smaller ones are clean samples, and the rest are noise samples.

[0113] Step 2.5, in addition, in order to improve the availability of noise samples, some of the noise samples are corrected. In order to avoid introducing more noise during correction, the correction confidence of each sample is calculated, as shown in formula (9):

[0114] (9);

[0115] in, It means to find the maximum value of sample gradient. It means to find the minimum value of sample gradient. The higher the correction confidence, the greater the possibility that the sample is corrected.

[0116] When the correction confidence meets the specified threshold condition, the noisy labeled sample will be assigned a pseudo label, that is, the category with the largest gradient value, as shown in formula (10):

[0117] (10);

[0118] in, Indicates the category index corresponding to the maximum gradient value; The value representing the pseudo label.

[0119] Furthermore, in step 3, dynamic partitioning and robust learning, the screened and corrected dataset is dynamically divided into three subsets: clean, noisy, and difficult, and different robust loss functions are applied to each subset to mitigate the impact of noisy labels, specifically:

[0120] In order to prevent the model from overfitting to data with noisy labels, the present invention further proposes a dynamic partitioning and robust learning strategy based on screening confidence and correction confidence. The multimodal dataset is divided into three subsets: a clean subset, a noisy subset, and a difficult subset. Since the noisy labels still exist in each subset with different proportions, different robust loss functions are used on each subset to mitigate the impact of noise.

[0121] Step 3.1, first of all, the selection and learning of the clean subset. According to the analysis in step 2.4, the smaller the L2 norm of the gradient, the higher the probability that the sample is clean. Therefore, this method selects the clean subset as:

[0122] (11);

[0123] in, represents small batches of data, Represents the i-th modal data; e i = sort(G( M ˜ i ))[| M ˜ i |*R(t)] represents the screening threshold of the i-th mode, represents the percentage of clean samples selected from the small batch; represents the function described in formula (8), Represents a sorting function.

[0124] In addition, to ensure that most samples remain in the clean subset to dominate model training before overfitting, this method proposes a dynamic selection strategy. It is initially set to a large value, and then gradually decreases with the training rounds until it stabilizes. Since the clean subset C has a more accurate label, the cross entropy loss is used for optimization of the clean subset, as shown in formula (12):

[0125] (12);

[0126] in, Represents the size of the clean subset C; Represents the loss value obtained by the cross entropy loss function for the data in the clean subset.

[0127] Step 3.2, followed by the selection and learning of noise subsets. In order to utilize the noise data outside the clean subset, this method groups the samples whose correction confidence is greater than the correction threshold into noise subsets. , as shown below:

[0128] (13)

[0129] in, Indicates the correction threshold, the value range is between 0-1; Indicates samples whose correction confidence is higher than the correction threshold; represents the pseudo label corrected according to formula (10); Represents mini-batches of data.

[0130] Since the corrected pseudo-labels may still be inaccurate, the model will be disturbed by inaccurate labels if it is directly optimized using cross entropy loss. Therefore, this method adopts a more robust loss, namely generalized cross entropy loss, which combines the advantages of mean absolute error (MAE) and cross entropy (CE) and is defined on the noisy subset. The generalized cross entropy loss function is shown in formula (14):

[0131] (14);

[0132] in, represents the noise subset; r ∈ （ 0 ， 1 ] , represents the adjustment parameter, which is used to balance robustness and learning efficiency. Close to 0, use L'Oreal ( ) law, the generalized cross entropy loss is equivalent to the cross entropy, and as it approaches 1, it will approach the mean absolute error.

[0133] Step 3.3, finally, the selection and learning of the difficult subset. After the above screening, the clean and noisy samples are excluded, and the remaining samples in the multimodal dataset are regarded as difficult samples. .

[0134] The samples in the difficult subset are ambiguous and difficult to correct. In order to avoid adding external noise, this method proposes to learn the common distribution of prediction probabilities from different modalities of paired samples, that is, distribution consistency learning. Therefore, for difficult samples, a distribution consistency learning strategy is proposed, using the mean squared error (MSE) loss function to align the prediction distributions between different modalities of samples.

[0135] Since this property is applicable to all samples, it is extended to the entire multimodal dataset as shown in formula (15):

[0136] (15);

[0137] in, Indicates the batch size; Represents the predicted probability distribution of other modal data except the current i-th modality, and stipulates ≠ ; Represents the value of the mean squared error loss function.

[0138] Furthermore, the step 4, cross-modal consistency learning, adopts a contrastive learning strategy based on dynamic partitioning and robust learning to ensure that the representations of different modal samples of the same target in the common feature space are close to each other, while samples of different targets should be far away, specifically:

[0139] In order to ensure that the embeddings of paired samples from different modalities are well aligned in the common feature space, in addition to the prediction label probability distribution consistency learning in step 3, this method also proposes a cross-modal consistency learning strategy. Through contrastive learning, the common semantic representations of different modalities of the same target are used as positive sample pairs. The distribution of positive samples should be close to each other, while other samples as negative samples should be far away. Therefore, the multimodal contrastive learning loss function is shown in formula (16):

[0140] (16);

[0141] in, is the temperature parameter; Indicates The embedding representation of the jth sample in the modality in the common semantic space.

[0142] Furthermore, in step 5, an overall objective function is constructed, and the overall objective function is formed by combining formula (12), formula (14), formula (15) and formula (16), and the performance of the model is optimized by minimizing the overall objective function, specifically:

[0143] In order to make the model converge gradually, in the initial stage, a training warm-up process is used, namely, cross entropy loss to realize the calculation of screening confidence and correction confidence, as shown in formula (5). Through warm-up, the screening and correction of noise samples are realized. Combining three different types of robust loss functions, the overall objective function of this method is shown in formula (17):

[0144] (17);

[0145] in, It is a balance parameter that adjusts the weights of the three loss functions.

[0146] By minimizing the overall objective function, the model can be iteratively optimized using the stochastic gradient descent optimization algorithm to achieve the optimization goal.

[0147] Finally, the data of another modality is retrieved through the queried image and text, and the results are presented in Table 2, Table 3 and Table 4 as mean average precision (mAP). The present invention is always better than other methods in most cases, especially in high-noise scenarios.

[0148] Table 2: mAP of cross-modal retrieval results with noisy labels on the Wikipedia dataset

[0149]

[0150] Table 3. mAP of cross-modal retrieval results with noisy labels on the PascalSentence dataset

[0151]

[0152] Table 4: mAP of cross-modal retrieval results with noisy labels on the XMediaNet dataset

[0153]

[0154] From the results in Tables 2, 3 and 4, it can be seen that the method proposed in the present invention performs well in cross-modal retrieval applications and outperforms existing methods in most cases. In particular, in scenarios with high noise ratios, the method of the present invention shows significant advantages. Therefore, from the overall analysis of the application example effect, the method proposed in the present invention is effective in making full use of multimodal data with noisy labels for cross-modal retrieval, and has achieved better cross-modal retrieval results than previous methods.

[0155] In summary, the present invention is a cross-modal retrieval method with noisy labels based on gradient-guided dynamic partitioning, which can effectively divide data subsets based on the gradient of the sample loss function and adopt different processing methods, thereby improving the learning performance and retrieval ability of the cross-modal retrieval model with noisy labels, and solving the problems of inaccurate labeling and high annotation cost of labeled data in multimodal data sets.

[0156] The above-mentioned examples only express the implementation modes of the present invention, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A cross-modal retrieval method with noisy labels based on gradient-guided dynamic partitioning, characterized in that: The cross-modal retrieval method with noisy labels comprises the following steps: Step 1: Common semantic representation learning, specifically: project each modality into the common semantic space and calculate the class prototype probability of the sample to achieve correct classification of the sample; Step 2: Gradient-guided screening and correction. Specifically, for the data after learning the common semantic representation, use the gradient information to screen out the clean label data and correct some of the noisy label data, thereby improving the quality of the dataset. Step 3: Dynamic partitioning and robust learning. Specifically, the dataset screened and corrected in step 2 is dynamically partitioned into three subsets: clean, noisy, and difficult, and different robust loss functions are applied to each subset. Step 4: cross-modal consistency learning, specifically: based on the dynamic partitioning and robust learning in step 3, a contrastive learning strategy is adopted to ensure that the representations of samples of different modalities of the same target in the common feature space are close to each other, while samples of different targets are far away from each other; Step 5: Construct an overall objective function and optimize the model performance by minimizing the overall objective function.

2. According to claim 1, a noisy label cross-modal retrieval method based on gradient-guided dynamic partitioning is characterized in that: The step 1 is specifically as follows: Step 1.1, let represents a multimodal dataset with m modalities, represents the i-th mode, each mode contains N samples; among them, represents the jth sample of the i-th mode, Representation sample The label of, where K represents the number of semantic categories, the corresponding one-hot vector is expressed as ; The modality includes picture or video modality; In view of the heterogeneous differences between different modalities, the pre-trained CLIP is used as the backbone network. The features extracted by the backbone network are used to learn the modality-specific transformation function, and the image or video modality is projected into a common semantic space. Therefore, the process of extracting features from each sample through the backbone network is shown in formula (1): （1）； in, Represent the transformation functions of different modes; Represents the multimodal features extracted by CLIP; Step 1.2, after obtaining the multimodal features extracted by CLIP, a three-layer fully connected network is stacked to project the extracted multimodal features into a common semantic space; as shown in formula (2): （2）； in, represents the feature extraction network of the i-th modality; represents the learnable parameter matrix; represents the dimension of the public semantic space, It means that the dimension of the result after the transformation of the formula is ; Represents the embedding representation of each sample in the common semantic space; Step 1.3, in the public semantic space, calculate the class prototype probability distribution of the sample to achieve the correct classification of the sample; for each category, calculate the average value of all its samples in the public semantic space, and use it as the class prototype of the category, and construct the class prototype as the anchor point of clustering ,in Represents the prototypes of k categories; calculate the similarity between each sample and each class prototype to achieve discriminative analysis of sample features, as shown in formula (3): （3）； in, Represents the similarity vector between the sample and each class prototype; The matrix representing all class prototype vectors connected in parallel is then transposed; Step 1.4, Sample The probability of belonging to each class is shown in formula (4): （4）； in, represents the temperature parameter, which is used to adjust the smoothness of the output probability; Representation sample The probability vector belonging to each class, each bit in the vector is the predicted probability value of the sample belonging to a certain category; By minimizing The learning is optimized by using the standard loss between the given labels.

3. According to claim 2, a noisy label cross-modal retrieval method based on gradient-guided dynamic partitioning is characterized in that: The step 2 is specifically as follows: Step 2.1: Use cross entropy loss to preliminarily calculate the gradient value distribution of all samples, and calculate the cross entropy loss function between the predicted probability value obtained by formula (4) and the one-hot vector of the given label. ; Step 2.2, for the cross entropy loss function The gradient calculation of is shown in formula (6): （6）； Step 2.3, for the sample The gradient is reflected in its similarity to the prototype of each class According to the given label gradient calculation, there are two cases, as shown in formula (7): （7）； Among them, cls represents the current sample label category. When the sample prediction label is the same as the true label, the corresponding gradient decreases by one; Step 2.4, calculate the L2 norm of the gradient as the screening confidence of the noisy sample, as shown in formula (8): （8）； in, Indicates L2 norm, filter confidence The smaller ones are clean samples, and the rest are noise samples; Step 2.5: In order to improve the availability of noise samples, some of the noise samples are corrected; in order to avoid introducing more noise during correction, the correction confidence of each sample is calculated, as shown in formula (9): （9）； in, It means to find the maximum value of sample gradient. Indicates the minimum value of the sample gradient; the higher the correction confidence, the greater the possibility that the sample is corrected; When the correction confidence meets the specified threshold condition, the noisy labeled sample will be assigned a pseudo label, the category with the largest gradient value, as shown in formula (10): （10）； in, Indicates the category index corresponding to the maximum gradient value; The value representing the pseudo label.

4. According to claim 3, a noisy label cross-modal retrieval method based on gradient-guided dynamic partitioning is characterized in that: In step 2.1, the cross entropy loss function between the predicted probability value obtained by formula (4) and the one-hot vector of a given label is calculated, as shown in formula (5): （5）； in, represents the unique hot vector corresponding to each sample label, m represents the number of modes, N represents the number of samples, Represents the value of the cross entropy loss function.

5. According to claim 3, a noisy label cross-modal retrieval method based on gradient-guided dynamic partitioning is characterized in that: In step 3, the multimodal dataset is divided into three subsets, namely, a clean subset, a noisy subset, and a difficult subset. Different robust loss functions are used on each subset to reduce the impact of noise; specifically: Step 3.1, first of all, the selection and learning of the clean subset. According to the analysis in step 2.4, the smaller the L2 norm of the gradient, the higher the probability that the sample is clean; the clean subset is selected as: （11）； in, represents small batches of data, Represents the i-th modal data; represents the screening threshold of the i-th mode, represents the percentage of clean samples selected from the small batch; represents the function described in formula (8), represents the sorting function; Using a dynamic selection strategy, Initially set to a large value, it gradually decreases with the training rounds until it stabilizes; the cross entropy loss function is used for the clean subset Optimize Step 3.2, followed by the selection and learning of noise subsets, grouping samples whose correction confidence is greater than the correction threshold into noise subsets , as shown below: （13）； in, represents the correction threshold; Indicates samples whose correction confidence is higher than the correction threshold; represents the pseudo label corrected according to formula (10); Represents small batches of data; Using generalized cross entropy loss function Define the loss on the noisy subset; Step 3.3, finally, the selection and learning of the difficult subset. ; For the difficult subset, a distribution consistency learning strategy is proposed, which uses the mean square error loss function to align the prediction distributions of different modalities of samples in the difficult subset. The mean square error loss function is adopted and extended to the entire multimodal dataset, as shown in formula (15): （15）； in, Indicates the batch size; Represents the predicted probability distribution of other modal data except the current i-th modality, and stipulates ≠ ; Represents the value of the mean squared error loss function.

6. According to claim 5, a noisy label cross-modal retrieval method based on gradient-guided dynamic partitioning is characterized in that: In step 3.1, the clean subset is optimized using the cross entropy loss function, as shown in formula (12): （12）； in, Represents the size of the clean subset C; Represents the loss value obtained by the cross entropy loss function for the data in the clean subset.

7. According to claim 5, a noisy label cross-modal retrieval method based on gradient-guided dynamic partitioning is characterized in that: In step 3.1, the threshold value is corrected The value range is between 0-1.

8. According to claim 5, a noisy label cross-modal retrieval method based on gradient-guided dynamic partitioning is characterized in that: In step 3.2, the generalized cross entropy loss function is shown in formula (14): （14）； in, represents the noise subset; , represents the adjustment parameter.

9. The method for cross-modal retrieval with noisy labels based on gradient-guided dynamic partitioning according to claim 5, characterized in that: The step 4 is specifically as follows: A cross-modal consistency learning strategy is proposed. Through contrastive learning, the common semantic representations of different modalities of the same target are used as positive sample pairs. The distribution of positive samples should be close to each other, while other samples as negative samples should be far away. Therefore, the multimodal contrastive learning loss function , as shown in formula (16): （16）； in, is the temperature parameter; Indicates The embedding representation of the jth sample in the modality in the common semantic space.

10. The method for cross-modal retrieval with noisy labels based on gradient-guided dynamic partitioning according to claim 9, characterized in that: The step 5 is specifically as follows: The overall objective function is the cross entropy loss function in step 3.

1. , the generalized cross entropy loss function in step 3.2 , the mean square error loss function in step 3.3 , multimodal contrastive learning loss function in step 4 constitute; In the initial stage, a training warm-up process is adopted, and the cross entropy loss function is used to calculate the screening confidence and correction confidence. Through warm-up, the screening and correction of noise samples are realized. Combining three different types of loss functions, the overall objective function is shown in formula (17): （17）； in, It is a balance parameter that adjusts the weights of the three loss functions; By minimizing the overall objective function, the stochastic gradient descent optimization algorithm is used for iterative optimization to obtain the optimized overall objective function, and then cross-modal cross retrieval is performed.

Images