A robust cross-modal retrieval method based on alignment self-correction

Abstract

The present invention discloses a robust cross-modal retrieval method based on alignment self-correction. The method comprises the following steps: extracting visual and textual features; inputting the visual and textual features into a cross-modal retrieval network for deep fusion; preheating the model using a multi-task training model; calculating the loss of all sample pairs using the preheated model; fitting a beta mixture model to the loss values; self-correcting the aligned labels in the data using the beta mixture model; and training the model using new labels until convergence to obtain a cross-modal retrieval model. The robust cross-modal retrieval method proposed in the present invention can adaptively correct erroneous data in cross-modal retrieval through loss distribution, thereby improving cross-modal retrieval performance under noise supervision.

Description

Technical Field

[0001] The present invention relates to the field of multimedia information processing, and in particular to a robust cross-modal retrieval method based on alignment self-correction. Background Art

[0002] Cross-modal retrieval involves searching for semantically matching samples in another modality, given a query term in one modality. With the rapid development of internet technology, the ubiquity of smart devices, and the popularity of mobile apps such as shopping and communication software, multimedia data is exploding. This surge in multimedia data has led to a significant demand for cross-modal retrieval. Currently, the need to search for desired products using text, as well as images and videos, is quite common. Research has found that current methods significantly degrade performance when training data contains noise, such as when images and text do not match.

[0003] Cross-modal retrieval can generally be achieved in two ways. The first is to minimize the similarity between negative examples and maximize the similarity between positive examples through contrastive learning. However, when the positive examples are mismatched samples, the model will still mistakenly maximize the similarity between positive examples. The second is to perform binary classification through image-text matching tasks, classifying matching samples as 1 and mismatched samples as 0. However, incorrect labels can also reduce the training effect of the model. Therefore, it is necessary to design a robust cross-modal retrieval method that can still maintain good retrieval performance in the presence of noisy data. Summary of the Invention

[0004] The purpose of this invention is to propose a robust cross-modal retrieval method based on alignment self-correction, combine multi-task training to improve model robustness, use Beta mixture distribution to model sample loss and correct labels to meet the needs of robust model training in the presence of noisy data.

[0005] The technical solution to achieve the purpose of the present invention is as follows: In a first aspect, the present invention provides a robust cross-modal retrieval method based on alignment self-correction, comprising the following steps:

[0006] Step 1: Extract visual features and text features;

[0007] Step 2: Input the visual features and text features into the cross-modal retrieval network for deep fusion;

[0008] Step 3: Use the multi-task training model for warm-up;

[0009] Step 4: Use the preheated model to calculate the loss of all sample pairs;

[0010] Step 5: Fit the beta mixture model to the loss value; continue training the cross-modal retrieval model, calculate the posterior probability that the sample loss value belongs to the beta mixture model with a larger mean, and self-correct the aligned labels;

[0011] Step 6: Use the Beta mixture model to self-correct the aligned labels in the data;

[0012] Step 7: Use the new labels to train the model until convergence to obtain a cross-modal retrieval model.

[0013] In a second aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the program.

[0014] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] Compared with the existing technology, the present invention has the following significant advantages: (1) the present invention uses multi-task training to effectively alleviate the problem of the Transformer model overfitting to noise too quickly, and expands the time window for label correction; (2) the present invention uses the Beta mixture model to adaptively cluster and correct labels for data in cross-modal retrieval, thereby improving the retrieval performance of the model under the supervision of noisy data.

[0016] The present invention will be described in further detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the robust cross-modal retrieval method based on alignment self-correction of the present invention.

[0018] Figure 2 This is the network framework diagram of the robust cross-modal retrieval method based on alignment self-correction in the present invention.

[0019] Figure 3 This is a visualization of the beta mixture model fitted on the Flickr30K dataset with 40% noise.

[0020] Figure 4 This is a visualization diagram of the training set of the present invention on the Flickr30K dataset with 40% noise. DETAILED DESCRIPTION

[0021] like Figure 1 、 Figure 2 As shown in the figure, a robust cross-modal retrieval method based on alignment self-correction is proposed to extract visual features and text features; the visual features and text features are input into the cross-modal retrieval network T θDo deep fusion in the training; use the multi-task training model for warm-up; use the warm-up model T θ Calculate the loss of all sample pairs; fit the loss value to the beta mixture model B; use the beta mixture model to self-correct the aligned labels in the data; use the new label training model until convergence to obtain a cross-modal retrieval model. The present invention uses the Transformer model as the basic model to fully integrate the modalities, combines multi-task training with image noise label learning technology, and improves the performance of the cross-modal retrieval model under noise interference. The steps of the present invention are described in detail below:

[0022] Step 1: Extract visual features and text features:

[0023] For visual text samples (I i ,T i ), use Faster R-CNN network to extract visual modality feature vector v = [v1,…v m ], Bert network model extracts text feature vector t=[t1,…t n ].

[0024] Step 2: Preheat using the multi-task training model:

[0025] Input visual features and text features into the cross-modal retrieval network T θ The specific process of deep fusion is: concatenate visual features and text features [v1,…v m ,t1,…t n ], after splicing, input the cross-modal retrieval Transformer network T θ Deep integration.

[0026] Step 3: Preheat using the multi-task training model:

[0027] (1) Transformer network T θ The output global [CLS] feature is classified into two categories, the cross entropy loss is calculated, and the image-text matching task is performed. The specific form is:

[0028]

[0029]

[0030] Among them, Classifier(·) is a two-layer neural network, l BCE is the binary classification loss, is the classification prediction of the global [CLS] feature, y i is the alignment label of the current image text pair, y i ∈{0,1}.

[0031] (2) Transformer network T θ Output <mask>The features are used for mask prediction, classification loss is calculated, and word mask prediction tasks are performed. The specific form is:

[0032] l MLM =-E (t,v)～D log P θ (t m |t \m ,v)

[0033] Among them, E(·) is the mean, (t, v)~D are the image text pair features (t, v) obtained by sampling the data in the dataset D, l MLM is the mask prediction loss, P θ (t m |t \m ,v) is the observed <mask>The word t \m After comparing with all image regions v <mask>Word prediction.

[0034] Step 4: Use the preheated model T θ Calculate the loss for all sample pairs:

[0035] Without updating the model, use the preheated model T θ Calculate the binary classification loss of the image-text matching task for all sample pairs.

[0036] Step 5. Fit the beta mixture model B to the loss value:

[0037] (1) First, a beta mixture model (BMM) with two components is established:

[0038]

[0039] Where p(l) is the probability of loss value l, μ k is the mixing coefficient of the kth mixing component, f(l|α k ,β k ) is the probability density function of the k-th Beta mixture distribution, that is:

[0040]

[0041] where Γ(·) is the gamma function, α k ,β k is the parameter of the kth Beta distribution and α k ,β k >0;

[0042] (2) Use the expectation maximization algorithm (EM) to fit the loss value to obtain the BMM model parameters:

[0043] Let the latent variable λ k (l) = p(k|l) represents the posterior probability that the loss value l belongs to the kth mixture component.

[0044] E-step: Fixed parameter μ k ,α k ,β k , use Bayes' rule to update the latent variables:

[0045]

[0046] M-step: fixed λ k (l), estimated distribution parameter α k ,β k :

[0047]

[0048] The E step is the expectation step, and the M step is the maximization step;

[0049] in Represents each sample The weighted average loss of Represents the weighted variance value:

[0050]

[0051]

[0052] Estimated mixture component coefficient μ k :

[0053]

[0054] Where N is the number of samples.

[0055] Step 6: Use the Beta mixture model to self-correct the aligned labels in the data:

[0056] y iref =H((1-λ t (l i ))y i +λ t (l i )z i )

[0057] Among them, z i is the category prediction of the model, H(·) is the category label of the class with the largest probability value, is the corrected label, λ t (l i )The posterior probability that the loss value l belongs to the noise mixture component, and we believe that the mixture component with a larger mean is the noise mixture component.

[0058] Step 7: Use the new labels to train the model until convergence to obtain a cross-modal retrieval model:

[0059]

[0060] Among them, l ref The adjusted loss.

[0061] Table 1 Comparison of methods on the Flickr30K dataset with 40% noise

[0062]

[0063]

[0064] Table 1 compares the results of our method, abbreviated as RCAR, with other online methods on the Flickr30K dataset containing 40% noise. The evaluation metrics include Image2Text (text retrieval using images), Text2Image (image retrieval using text), and R@k (the fraction of correctly retrieved terms among the K closest points to the query). The "↑" symbol indicates higher performance. Our method achieves the highest ranking across all six metrics, demonstrating its effectiveness in training robust cross-modal retrieval models.

[0065] Figure 3 This is a visualization of the beta mixture model fitted on the Flickr30K dataset with 40% noise. It can be found that the beta mixture model fits the loss distribution of both clean and noisy samples well. Figure 4 The feature distribution diagram of the training set of the Flickr30K dataset with 40% noise of the present invention is shown. It can be found that the noise samples are mostly concentrated in the same area as the negative examples, and the model does not overfit the noise samples.

[0066] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.< / mask> < / mask> < / mask>

Claims

1. A robust cross-modal retrieval method based on alignment self-correction, characterized by: The steps include: Step 1: Extract visual features and text features; Step 2: Concatenate visual features with text features [v1,…v m ,t1,…t n ], after splicing, input the cross-modal retrieval Transformer network T θ Deep integration; Step 3: Use the multi-task training model for warm-up; multi-task training includes the following parts: (1) Transformer network T θ The output global [CLS] feature is classified into two categories, the cross entropy loss is calculated, and the image-text matching task is performed. The specific form is: Among them, Classifier(·) is a two-layer neural network, l BCE is the binary classification loss, is the classification prediction of the global [CLS] feature, y i is the alignment label of the current image text pair, y i ∈{0,1}; (2) Transformer network T θ Output <mask> The features are used for mask prediction, classification loss is calculated, and word mask prediction tasks are performed. The specific form is:< / mask> l MLM =-E (t,v)～D logP θ (t m |t \m ,v) Among them, E(·) is the mean, (t, v)~D is the image text pair feature (t, v) obtained by sampling the data in the dataset D, l MLM is the mask prediction loss, P θ (t m |t \m ,v) is the observed <mask>The word t \m After comparing with all image regions v <mask> Word prediction;< / mask> < / mask> Step 4: Use the preheated model to calculate the loss of all sample pairs; Step 5: Fit the beta mixture model to the loss value; continue training the cross-modal retrieval model, calculate the posterior probability that the sample loss value belongs to the beta mixture model with a larger mean, and self-correct the aligned labels; Step 6: Use the Beta mixture model to self-correct the aligned labels in the data; Step 7: Use the new labels to train the model until convergence to obtain a cross-modal retrieval model.

2. The robust cross-modal retrieval method based on alignment self-correction according to claim 1, characterized in that: In step 1, the specific process of extracting visual features and text features is as follows: for the visual text sample (I i ,T i ), use Faster R-CNN network to extract visual modality feature vector v = [v1,…v m ], Bert network model extracts text feature vector t=[t1,…t n ].

3. The robust cross-modal retrieval method based on alignment self-correction according to claim 1, characterized in that: In step 4, without updating the model, the preheated model T is used. θ Calculate the binary classification loss of the image-text matching task for all sample pairs.

4. The robust cross-modal retrieval method based on alignment self-correction according to claim 1, characterized in that: In step 5, the specific process of fitting the beta mixture model BMM to the loss value is as follows: (1) First, establish a beta mixture model with two components: in is the loss value Probability of occurrence, μ k is the mixing coefficient of the kth mixing component, is the probability density function of the kth Beta mixture distribution, that is: where Γ(·) is the gamma function, α k ,β k is the parameter of the kth Beta distribution and α k ,β k >0; (2) Use the expectation maximization algorithm to fit the loss value to obtain the BMM model parameters: Set the latent variable Indicates the loss value The posterior probability of belonging to the k-th mixture component; E-step: Fixed parameter μ k ,α k ,β k , use Bayes' rule to update the latent variables: M step: fixed Estimate the distribution parameter α k ,β k : in Represents each sample The weighted average loss of Represents the weighted variance value: Estimated mixture component coefficient μ k : Where N is the number of samples.

5. The robust cross-modal retrieval method based on alignment self-correction according to claim 1, characterized in that: In step 6, the beta mixture model is used to self-correct the alignment labels in the data. The specific form of the correction method is: Among them, z i is the category prediction of the model, H(·) is the category label of the class with the largest probability value, For the revised label, is the loss value The posterior probability of belonging to a noise mixture component.

6. The robust cross-modal retrieval method based on alignment self-correction according to claim 5, characterized in that: In step 7, the new label is used to train the model until convergence to obtain a cross-modal retrieval model. The specific form of the loss function is: in, The adjusted loss.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images