Self-discarding and dual-weighting method for noise correlation

Abstract

The invention relates to a self-discarding and dual-weighting method for noise correlation, and relates to the field of image-text matching. The method comprises the following steps: performing similarity calculation on an image-text pair input model in a noise data set, and selecting a similarity image-text pair as a clean data set; using the clean data set to construct evaluation entries for samples in the data set to obtain a memory bank; performing similarity calculation on image-text pairs in the noise data set, and selecting a proper threshold to discard samples with low similarity to generate a partial data set; according to the image-text pair similarity obtained through calculation, a Gaussian mixture model is used for fitting to obtain a confidence coefficient weight; creating a twinborn model by copying parameters of the parameter model, and training the image-text pairs in the partial data set by using the twinborn model; evaluating the performance change of the twin model before and after training by using a memory bank to obtain an importance weight; and weighting and training the data in the partial data set by the model by using a confidence coefficient weight and an importance weight.

Description

Technical Field

[0001] The invention relates to a self-discarding and dual-weight method for noise association, belonging to the technical field of image and text retrieval. Background Art

[0002] Given a pair of images and texts, this technique aims to index and obtain samples with similar semantic content in one modality through samples in another modality. Current image-text retrieval-related technologies have been successfully applied to tasks such as online public opinion event warning, multimedia event detection, and medical data classification. The core goal of image-text retrieval is to reveal and understand the deep-level associations between samples of different modalities, that is, to perform cross-modal matching. Traditional image-text retrieval models can be roughly divided into two categories: 1) global-level matching methods, which align visual features captured from images with overall semantic features extracted from text; and 2) local-level matching methods, which aim to establish fine-grained connections between local regions of images and individual words in text. In recent years, visual language pre-training models (such as CLIP) have demonstrated strong zero-shot learning capabilities. Furthermore, CLIP has shown its flexibility and can be widely used in a variety of cross-modal tasks, including detection, segmentation, and image description.

[0003] Although the above methods have achieved excellent performance in the field of image-text retrieval, they all have a premise that the data used for model training must be correctly corresponding. Existing methods require precisely annotated data pairs to achieve matching between the two modalities of image and text, but it is costly and time-consuming to obtain such data in real scenarios. Therefore, in actual training scenarios, there are often unmatched image-text pairs that are incorrectly annotated as matching image-text pairs, which leads to the problem of noise association. Unlike the noise label problem that has been widely explored before, the noise association problem refers to unmatched cross-modal pairs, which is an instance-level noise. Therefore, the noise association problem is often more difficult than the noise label problem. In previous studies, methods such as NCR, CREAM, and BiCro have explored the problem of noise association. However, in the field of visual language pre-training (VLP), such research is still in its infancy. The noise association problem of VLP is different from general models in two aspects: 1) Most previous methods use the method of starting from scratch to distinguish noise through the memory effect of DNN. However, for VLP, the cost of training from scratch is very high. Therefore, most of the previous methods are not applicable to VLP; 2) Most of the previous methods require warm-up to achieve the initial fitting of the model, but for VLP, the model itself is pre-trained. Therefore, if the warm-up method is used, VLP will quickly fit the noise, significantly reducing the performance of VLP. In view of the characteristics of VLP, subsequent studies have adopted a fine-tuning method to train the model. The DSDMR model uses the excellent zero-sample ability of the pre-trained model to distinguish between noise samples and clean samples. The NPC model allocates confidence weights by evaluating the negative impact of each sample.

[0004] Although these models have shown encouraging performance, they usually adopt a two-partition or three-partition approach. The two-partition divides the dataset into two subsets: a clean set and a noisy set. This division method is too rough, so subsequent work proposed a three-partition method to divide the dataset into three subsets: a clean set, a fuzzy set, and a noisy set. However, the three-partition method still ignores an important factor, that is, not all clean samples are equally important for VLP training. In addition, the above methods all utilize noisy data through weighting and other methods. However, simple weighting is difficult to completely eliminate the negative effects of noise. Therefore, it can be concluded that for VLP fine-tuning, a small amount of clean data is far more effective than a large amount of noisy data.

[0005] Based on this, the present application proposes a self-discarding and dual-weighting method for noise correlation to solve the VLP fine-tuning problem in noise correlation scenarios. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies described in the background technology and provide a self-discarding and dual-weighting method for noise association. The samples are divided into: clean and important samples, clean but unimportant samples, fuzzy samples and noise samples through a four-fold division strategy; most of the negative effects brought by noise samples are eliminated through a self-discarding strategy, thereby effectively improving the accuracy of the model; more reasonable weights are assigned to fuzzy samples through confidence weights and the small number of noise samples remaining after sub-discarding are further reduced; and the importance weights are used to make the model pay more attention to the important parts of clean samples, thereby ultimately achieving a four-fold division and utilization of samples.

[0007] In order to achieve the above object, the present invention is implemented by adopting the following technical solutions.

[0008] A self-discarding and dual-weighting method for noise correlation comprises the following steps:

[0009] Step 1: Input the image-text pairs in the noise data set into the model to calculate the similarity, and select the image-text pairs with high similarity as the clean data set;

[0010] Step 2: Use the clean dataset to construct evaluation entries for the samples in the dataset to obtain a memory library;

[0011] Step 3: Calculate the similarity of the image-text pairs in the noise data set, select an appropriate threshold to discard samples with low similarity to generate a partial data set;

[0012] Step 4: Use the Gaussian mixture model to fit the calculated image-text similarity to obtain the confidence weight;

[0013] Step 5: Create a twin model by copying the parameter model parameters, and use the twin model to train the image-text pairs in some data sets;

[0014] Step 6: Use the memory bank to evaluate the performance changes of the twin model before and after training to obtain the importance weight;

[0015] Step 7: The model uses confidence weights and importance weights to weight and train the data in some datasets.

[0016] As a preferred embodiment of the present invention, the specific process of step 1 is as follows:

[0017] Step 1-1, using the parameters of the pre-trained CLIP model, the image-text pairs in the noisy dataset input to the encoder are mapped into vector space to generate image features and text features respectively. The representation in the noisy dataset is as follows:

[0018]

[0019] Where N represents the number of image-text pairs in the dataset, (I i ,T i ) represents the i-th image-text pair in the data set;

[0020] Step 1-2, the cosine similarity calculation is performed on the features after the above conversion to generate a similarity matrix and the values ​​on the main diagonal of the matrix are used as the similarity of the image-text pair. This process is expressed as:

[0021]

[0022] Where f(I i ) indicates picture I i The image features obtained by the image encoder f, g(T i ) represents the text T i The text features obtained by the text encoder g, ||·|| represents the modulus of the feature vector, S(I i ,T i ) and s i,i Both represent the cosine similarity of the image-text pair;

[0023] Step 1-3: Select a high threshold to filter out image-text pairs with high similarity to generate a clean data set. The clean data set is represented as:

[0024] D c ={(I i ,T i )|s i,i >β}, (3)

[0025] Where D c represents a clean data set, and β represents the threshold for screening high-similar image-text pairs.

[0026] As a preferred embodiment of the present invention, the specific process of step 2 is as follows:

[0027] Step 2-1: Compare the images in each image-text pair in the dataset D with the images in the clean dataset D c Calculate the similarity of the image-text pairs in the image, and select the image-text pair with the highest similarity as one of the evaluation items;

[0028] Step 2-2: Compare the text in each image-text pair in dataset D with the clean dataset D c The text similarity of the image-text pair in the image-text pair is calculated, and the image-text pair with the highest similarity is selected as another item of the evaluation item;

[0029] Step 2-3: Build a memory bank for each pair of image-text pairs in the data set. The memory bank is represented as:

[0030]

[0031] Where D MB Represents the memory bank, (I i I ,T i I ) represents the clean data set D c The image-text pair in the dataset D that maintains the highest similarity with the image-text pair in the dataset D, (I i T ,T i T ) represents the clean data set D c The image-text pair in which the text maintains the highest similarity with the text of an image-text pair in dataset D.

[0032] As a preferred solution of the present invention, the specific process of step 3 is:

[0033] Step 3 selects an appropriate threshold to discard samples with low similarity to generate a partial data set, which is represented by:

[0034] D p ={(I i ,T i )|s i,i >α}, (5)

[0035] Where D p Represents part of the data set, and α represents the threshold for dividing low-similarity samples.

[0036] As a preferred solution of the present invention, the specific process of step 4 is:

[0037] The formula for generating confidence weights using the Gaussian mixture model described in step 4 is:

[0038]

[0039] Among them, α k represents the coefficient of the mixture component, represents the probability density of the kth component, θ k Represents a Gaussian component with a higher mean.

[0040] As a preferred solution of the present invention, the specific process of step 5 is:

[0041] In step 5, the twin model is used to train the image-text pairs in some data sets, and the InfoNCE loss of the image-text pairs is calculated as:

[0042]

[0043] in represents the InfoNCE loss, represents the cross entropy loss, and τ represents the temperature coefficient.

[0044] As a preferred solution of the present invention, the specific process of step 6 is:

[0045] In step 6, the memory bank is used to evaluate the performance change of the twin model before and after training to obtain the importance weight. The performance is expressed in the form of loss, and the formula is:

[0046]

[0047] in represents the image-to-text loss calculated for the i-th sample pair at the t-th epoch, The text-to-image loss calculated for the i-th sample pair at the t-th epoch, (I i I ,T i I ) represents the evaluation entry in the memory bank that has the highest similarity with the sample image, (I i T ,T i T ) represents the evaluation entry in the memory bank that has the highest similarity with the sample text.

[0048] The formula for calculating the performance change before and after training is:

[0049]

[0050] where c i Represents the performance change of the i-th sample before and after training.

[0051] The model uses a mapping function to calculate the importance weight of sample pairs, and the formula is:

[0052]

[0053] in represents the importance weight of the i-th sample pair, where q(·) can be tanh(·), sigmoid(·), etc.

[0054] As a preferred solution of the present invention, the specific process of step 7 is:

[0055] In step 7, the model uses confidence weights and importance weights to weight and train the data in part of the data set. The loss function of the training is:

[0056] L=w con w sig L InfoNCE , (14)

[0057] Where L is the overall loss function, w con is the confidence weight of the sample pair, w sig is the importance weight of the sample pair, L InfoNCE Loss for InfoNCE.

[0058] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0059] 1. The present invention proposes a self-discarding and dual-weighting method for noise association. To address the noise association problem in actual labeled data, self-discarding is proposed to discard most of the noise samples, thereby improving the robustness of the image-text matching model to noise and significantly improving the accuracy of image-text matching.

[0060] 2. The present invention further weights the self-discarded samples through double weights, thereby improving the reliability of the samples, using confidence weights to weaken the negative impact of residual noise samples, more reasonably utilizing fuzzy samples, and using importance weights to make the model pay more attention to the important parts of clean samples, thereby achieving robust learning of noisy data. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a flowchart of self-discarding and dual weighting for noise correlation of the present invention;

[0062] Figure 2 It is a schematic diagram of the overall process of self-discarding and dual weighting for noise correlation in the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the attached Figure 1 The method of the present invention is described in detail with specific embodiments.

[0064] It is worth mentioning that the embodiments of the present invention are described in detail below with reference to the accompanying drawings, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0065] With the increasing maturity of visual semantic pre-training (VLP), image-text retrieval related technologies have been successfully applied to tasks such as network public opinion event warning, multimedia event detection, and medical data classification. However, there are noise correlation problems in the data and the distribution is relatively complex. The existing methods for noise correlation cannot be well applied to the VLP field. Studies have shown that for VLP fine-tuning, a small amount of clean samples is far more effective than a large amount of noisy data. Based on this, the present invention proposes a self-discarding and dual weighting method for noise correlation.

[0066] like Figure 1As shown, the present invention proposes a UAV target visual positioning method based on multi-source priors, comprising the following steps:

[0067] Step 1: Input the image-text pairs in the noise data set into the model to calculate the similarity, and select the image-text pairs with high similarity as the clean data set.

[0068] The specific process is:

[0069] Step 1-1, using the parameters of the pre-trained CLIP model, the image-text pairs in the noisy dataset input to the encoder are mapped into vector space to generate image features and text features respectively. The representation in the noisy dataset is as follows:

[0070]

[0071] Where N represents the number of image-text pairs in the dataset, (I i ,T i ) represents the i-th image-text pair in the data set;

[0072] Step 1-2, the cosine similarity calculation is performed on the features after the above conversion to generate a similarity matrix and the values ​​on the main diagonal of the matrix are used as the similarity of the image-text pair. This process is expressed as:

[0073]

[0074] Where f(I i ) indicates picture I i The image features obtained by the image encoder f, g(T i ) represents the text T i The text features obtained by the text encoder g, ||·|| represents the modulus of the feature vector, S(I i ,T i ) and s i,i Both represent the cosine similarity of the image-text pair;

[0075] Step 1-3: Select a high threshold to filter out image-text pairs with high similarity to generate a clean data set. The clean data set is represented as:

[0076] D c ={(I i ,T i )|s i,i >β}, (3)

[0077] Where D c represents a clean data set, and β represents the threshold for screening high-similar image-text pairs.

[0078] Step 2: Use the clean dataset to construct evaluation entries for the samples in the dataset to obtain a memory library.

[0079] The specific process is:

[0080] Step 2-1: Compare the images in each image-text pair in the dataset D with the images in the clean dataset D c Calculate the similarity of the image-text pairs in the image, and select the image-text pair with the highest similarity as one of the evaluation items;

[0081] Step 2-2: Compare the text in each image-text pair in dataset D with the clean dataset D c The text similarity of the image-text pair in the image-text pair is calculated, and the image-text pair with the highest similarity is selected as another item of the evaluation item;

[0082] Step 2-3: Build a memory bank for each pair of image-text pairs in the data set. The memory bank is represented as:

[0083]

[0084] Where D MB Represents the memory bank, (I i I ,T i I ) represents the clean data set D c The image-text pair in the dataset D that maintains the highest similarity with the image-text pair in the dataset D, (I i T ,T i T ) represents the clean data set D c The image-text pair in which the text maintains the highest similarity with the text of an image-text pair in dataset D.

[0085] Step 3: Calculate the similarity of the image-text pairs in the noise data set, select an appropriate threshold to discard samples with low similarity to generate a partial data set. The partial data set is represented as:

[0086] D p ={(I i ,T i )|s i,i >α}, (5)

[0087] Where D p Represents part of the data set, and α represents the threshold for dividing low-similarity samples.

[0088] Step 4: Use the Gaussian mixture model to fit the calculated image-text pair similarity to obtain the confidence weight. The formula for generating the confidence weight using the Gaussian mixture model is:

[0089]

[0090] Among them, αk represents the coefficient of the mixture component, represents the probability density of the kth component, θ k Represents a Gaussian component with a higher mean.

[0091] Step 5: Create a twin model by copying the parameter model parameters, and use the twin model to train the image-text pairs in some data sets. The InfoNCE loss of the image-text pair is calculated as:

[0092]

[0093] in represents the InfoNCE loss, represents the cross entropy loss, and τ represents the temperature coefficient.

[0094] Step 6: Use the memory bank to evaluate the performance change of the twin model before and after training to obtain the importance weight. The performance is expressed in the form of loss, and the formula is:

[0095]

[0096] in represents the image-to-text loss calculated for the i-th sample pair at the t-th epoch, The text-to-image loss calculated for the i-th sample pair at the t-th epoch, (I i I ,T i I ) represents the evaluation entry in the memory bank that has the highest similarity with the sample image, (I i T ,T i T ) represents the evaluation entry in the memory bank that has the highest similarity with the sample text.

[0097] The formula for calculating the performance change before and after training is:

[0098]

[0099] where c i Represents the performance change of the i-th sample before and after training.

[0100] The model uses a mapping function to calculate the importance weight of sample pairs, and the formula is:

[0101]

[0102] in represents the importance weight of the i-th sample pair, where q(·) can be tanh(·), sigmoid(·), etc.

[0103] Step 7: The model uses confidence weights and importance weights to weight and train the data in some datasets. The loss function of the training is:

[0104]

[0105] in is the overall loss function, w con is the confidence weight of the sample pair, w sig is the importance weight of the sample pair, L InfoNCE Loss for InfoNCE.

[0106] The present invention discloses a self-discarding and dual-weighting method for noise association. Aiming at the noise association problem in actual labeled data, the present invention proposes self-discarding to discard most of the noise samples, thereby improving the robustness of the image-text matching model to noise and significantly improving the accuracy of image-text matching. The dual weights are used to further weight the samples after self-discarding to improve the reliability of the samples, and the confidence weights are used to weaken the negative impact of the residual noise samples, so that the fuzzy samples are used more reasonably. The importance weights are used to make the model pay more attention to the important parts of the clean samples, thereby realizing robust learning of the noise data.

[0107] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A self-discarding and dual-weighting method for noise correlation, characterized in that The steps include: Step 1: Input the image-text pairs in the noise data set into the model to calculate the similarity, and select the image-text pairs with high similarity as the clean data set; Step 2: Use the clean dataset to construct evaluation entries for the samples in the dataset to obtain a memory library; Step 3: Calculate the similarity of the image-text pairs in the noise data set, select an appropriate threshold to discard samples with low similarity to generate a partial data set; Step 4: Use the Gaussian mixture model to fit the calculated image-text similarity to obtain the confidence weight; Step 5: Create a twin model by copying the parameter model parameters, and use the twin model to train the image-text pairs in some data sets; Step 6: Use the memory bank to evaluate the performance changes of the twin model before and after training to obtain the importance weight; Step 7: The model uses confidence weights and importance weights to weight and train the data in some datasets.

2. A self-discarding and dual-weighting method for noise correlation according to claim 1, characterized in that: Step 1 specifically includes: Step 1-1, using the parameters of the pre-trained CLIP model, the image-text pairs in the noisy dataset input to the encoder are mapped into vector space to generate image features and text features respectively. The representation in the noisy dataset is as follows: Where N represents the number of image-text pairs in the dataset, (I i ,T i ) represents the i-th image-text pair in the data set; Step 1-2, the cosine similarity calculation is performed on the features after the above conversion to generate a similarity matrix and the values ​​on the main diagonal of the matrix are used as the similarity of the image-text pair. This process is expressed as: Where f(I i ) indicates picture I i The image features obtained by the image encoder f, g(T i ) represents the text T i The text features obtained by the text encoder g, ||·|| represents the modulus of the feature vector, S(I i ,T i ) and s i,i Both represent the cosine similarity of the image-text pair; Step 1-3: Select a high threshold to filter out image-text pairs with high similarity to generate a clean data set. The clean data set is represented as: D c ={(I i ,T i )|s i,i >β}, (3) Where D c represents a clean data set, and β represents the threshold for screening high-similar image-text pairs.

3. A self-discarding and dual-weighting method for noise correlation according to claim 1, characterized in that: Step 2 specifically includes: Step 2-1: Compare the images in each image-text pair in the dataset D with the images in the clean dataset D c Calculate the similarity of the image-text pairs in the image, and select the image-text pair with the highest similarity as one of the evaluation items; Step 2-2: Compare the text in each image-text pair in dataset D with the clean dataset D c The text similarity of the image-text pair in the image-text pair is calculated, and the image-text pair with the highest similarity is selected as another item of the evaluation item; Step 2-3: Build a memory bank for each pair of image-text pairs in the data set. The memory bank is represented as: Where D MB Represents the memory bank, Denotes the clean data set D c The image-text pair in which the image-text pair in the dataset D has the highest similarity. Denotes the clean data set D c The image-text pair in which the text maintains the highest similarity with the text of an image-text pair in dataset D.

4. A self-discarding and dual-weighting method for noise correlation according to claim 1, characterized in that: In step 3, an appropriate threshold is selected to discard samples with low similarity to generate a partial data set, which is represented as: D p ={(I i ,T i )|s i,i >α}, (5) Where D p Represents part of the data set, and α represents the threshold for dividing low-similarity samples.

5. A self-discarding and dual-weighting method for noise correlation according to claim 1, characterized in that: The formula for generating confidence weights using the Gaussian mixture model described in step 4 is: Among them, α k represents the coefficient of the mixture component, represents the probability density of the kth component, θ k Represents a Gaussian component with a higher mean.

6. A self-discarding and dual-weighting method for noise correlation according to claim 1, characterized in that: In step 5, the twin model is used to train the image-text pairs in some data sets, and the InfoNCE loss of the image-text pairs is calculated as: in represents InfoNCE loss, CE represents cross entropy loss, and τ represents temperature coefficient.

7. A self-discarding and dual-weighting method for noise correlation according to claim 1, characterized in that: In step 6, the memory bank is used to evaluate the performance change of the twin model before and after training to obtain the importance weight. The performance is expressed in the form of loss, and the formula is: in represents the image-to-text loss calculated for the i-th sample pair at the t-th epoch, The text-to-image loss calculated for the i-th sample pair at the t-th epoch, represents the evaluation entry in the memory bank that has the highest similarity with the sample image, Indicates the evaluation item in the memory bank that has the highest similarity with the sample text; The formula for calculating the performance change before and after training is: where c i Represents the performance change of the i-th sample before and after training; The model uses a mapping function to calculate the importance weight of sample pairs, and the formula is: in represents the importance weight of the i-th sample pair, where q(·) can be tanh(·), sigmoid(·), etc.

8. A self-discarding and dual-weighting method for noise correlation according to claim 1, characterized in that: In step 7, the model uses confidence weights and importance weights to weight and train the data in part of the data set. The loss function of the training is: in is the overall loss function, w con is the confidence weight of the sample pair, w sig is the importance weight of the sample pair, Loss for InfoNCE.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the self-discarding and dual-weight method for noise correlation described in any one of claims 1 to 8 is implemented.

Images