Model training method and device for long-tail noise, equipment and medium

Abstract

The invention relates to the technical field of computer vision, in particular to a model training method, device, equipment and medium for long-tail noise, and the method comprises the steps: inputting an input image, a text cue word and an observation label into a pre-trained visual language model, and setting a fine tuning module in an image encoder of the visual language model; obtaining text features, image features and original output values for each category in the visual language model, and obtaining a text prediction label; and determining a supervised start-stop state and a target loss function, and updating the fine tuning module based on the target loss function to obtain a trained visual language model. According to the method, whether text image alignment prior auxiliary supervision is needed or not is determined by evaluating the difference between the text prediction label and the observation label, and the classification accuracy of head class and tail class samples in a high-noise scene is improved by calibrating the deviation between the learned features and the observation label.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a model training method, device, equipment and medium for long-tail noise. Background Art

[0002] The field of computer vision usually trains models based on large-scale public datasets. However, in the real world, datasets often have class imbalance problems and exhibit the characteristics of long-tail distribution, that is, the head class contains most samples, while the tail class has insufficient samples. The dataset also has the problem of incorrect image labeling, called noisy labels, and it is expensive to create a balanced dataset with correctly labeled categories. Therefore, it is usually necessary to train models on datasets with both long-tail distribution and label noise. In order to solve these problems, the long-tail noise label learning method is introduced.

[0003] However, the existing long-tail noise label learning method ignores the impact of different noise rates in the data on model training. In high-noise environments, the long-tail noise label learning method will be insufficient because the noise labels will destroy the reliability of the samples and blur the distinction between noise samples and tail class samples, resulting in low classification accuracy of head class and tail class samples in high-noise scenarios.

[0004] Therefore, the prior art has defects and needs to be improved and developed. Summary of the invention

[0005] The present application provides a model training method, apparatus, device and medium for long-tail noise to solve the technical problem in the related art that the classification accuracy of head class and tail class samples is low in high-noise scenarios.

[0006] To achieve the above objectives, this application adopts the following technical solutions: A model training method for long-tail noise, wherein the method comprises: Inputting an input image in a target long-tail noise dataset, a text prompt word corresponding to the input image, and an observation label into a pre-trained visual language model, wherein the visual language model includes a text encoder and an image encoder, and the image encoder is provided with a fine-tuning module; In the visual language model, the text prompt word is processed by a text encoder to obtain text features, the input image is processed by an image encoder to obtain image features, and the input image is processed by the image encoder with a fine-tuning module to obtain original output values ​​for each category; Calculating the text-image similarity between the text feature and the image feature, and obtaining a text prediction label based on the text-image similarity; The supervision start and stop state is determined based on the text prediction label and the observation label, the target loss function is determined based on the supervision start and stop state and the original output value, and the fine-tuning module is updated based on the target loss function to obtain a visual language model that has completed training.

[0007] In one embodiment of the present application, determining the supervision start / stop state based on the text prediction label and the observation label includes: Calculating the overlap rate between the text prediction label and the observation label; Obtaining a preset control threshold, comparing the overlap rate with the control threshold, and obtaining a comparison result; The supervision start and stop status is determined according to the comparison result.

[0008] In one embodiment of the present application, determining the supervision start / stop state according to the comparison result includes: If the comparison result is that the overlap rate is greater than the control threshold, the supervision start-stop state is a supervision stop state; If the comparison result is that the overlap rate is less than or equal to the control threshold, the supervision start-stop state is a supervision enabled state.

[0009] In one embodiment of the present application, determining a target loss function based on the supervision start / stop state and the original output value includes: If the supervision start-stop state is the supervision enabled state, calculating information divergence based on the text image similarity and the original output value, and using the information divergence as a first loss function; Obtain an image prediction label based on the original output value, and calculate a second loss function according to the image prediction label and the observation label; A target loss function is obtained based on the first loss function and the second loss function.

[0010] In one embodiment of the present application, obtaining a target loss function based on the first loss function and the second loss function includes: Setting a first weight for the first loss function and a second weight for the second loss function, wherein the sum of the first weight and the second weight is 1; The first loss function and the second loss function are weightedly summed based on the first weight and the second weight to obtain a target loss function.

[0011] In one embodiment of the present application, determining the target loss function based on the supervision start / stop state and the original output value further includes: If the supervision start-stop state is a supervision disabled state, obtaining an image prediction label based on the original output value; A second loss function is calculated according to the image prediction label and the observation label, and the second loss function is used as the target loss function.

[0012] In one embodiment of the present application, obtaining an image prediction label based on the original output value includes: The original output value is converted into a probability distribution of each category using an activation function, and an image prediction label is obtained according to the probability distribution.

[0013] The present application also provides a model training device for long-tail noise, wherein the device comprises: An image input module is used to input an input image in a target long-tail noise dataset, a text prompt word corresponding to the input image, and an observation label into a pre-trained visual language model, wherein the visual language model includes a text encoder and an image encoder, and the image encoder is provided with a fine-tuning module; A feature extraction module, used for obtaining text features after the text prompt word is processed by a text encoder, obtaining image features after the input image is processed by an image encoder, and obtaining original output values ​​for each category after the input image is processed by the image encoder with a fine-tuning module in the visual language model; A text prediction module, used to calculate the text-image similarity between the text feature and the image feature, and obtain a text prediction label based on the text-image similarity; An updating module is used to determine a supervision start / stop state based on the text prediction label and the observation label, determine a target loss function based on the supervision start / stop state and the original output value, and update the fine-tuning module based on the target loss function to obtain a visual language model that has completed training.

[0014] The present application also provides a device, which includes: a memory, a processor, and a model training program for long-tail noise stored in the memory and executable on the processor, wherein the model training program for long-tail noise implements the steps of the model training method for long-tail noise as described above when executed by the processor.

[0015] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the model training method for long-tail noise as described above.

[0016] Beneficial effects of the present invention: The method of the embodiment of the present invention inputs the input image in the target long-tail noise data set, the text prompt word corresponding to the input image and the observation label into the pre-trained visual language model, the visual language model includes a text encoder and an image encoder, and the image encoder is provided with a fine-tuning module; in the visual language model, the text prompt word is processed by the text encoder to obtain text features, the input image is processed by the image encoder to obtain image features, and the input image is processed by the image encoder with a fine-tuning module to obtain the original output value for each category; the text image similarity between the text feature and the image feature is calculated, and the text prediction label is obtained based on the text image similarity; the supervision start and stop state is determined based on the text prediction label and the observation label, the target loss function is determined based on the supervision start and stop state and the original output value, and the fine-tuning module is updated based on the target loss function to obtain a visual language model that has been trained. This application determines whether text image alignment prior auxiliary supervision is required by evaluating the difference between the text prediction label and the observation label, and improves the classification accuracy of head and tail class samples in high noise scenes by calibrating the deviation between the learned features and the observation label. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a preferred embodiment of the model training method for long-tail noise in the present invention.

[0018] Figure 2 It is a model logic block diagram of the model training method for long-tail noise in the present invention.

[0019] Figure 3 It is a functional principle block diagram of a preferred embodiment of the model training device for long-tail noise in the present invention.

[0020] Figure 4 It is a functional principle block diagram of a preferred embodiment of the device in the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] Long-tail noisy label learning methods can be divided into the following categories: (1) emphasizing the importance of different samples by reweighting or regularization; (2) selecting clean samples according to specific criteria; (3) developing improved representation learning methods. The above methods can effectively improve the robustness of the model on long-tail noisy label data. However, they ignore the impact of different noise ratios on model training, resulting in poor performance in high-noise scenarios. It is worth noting that low levels of label noise have relatively little impact on model performance. Therefore, it is necessary to deal with high-noise ratio scenarios in a targeted manner, where unreliable labels constitute one of the main problems in noisy label learning for long-tail data. In this case, noisy labels introduce a large amount of misleading supervisory signals, which in turn leads to the inability to effectively distinguish between noisy samples and clean samples, or to effectively improve feature representation, which leads to the accumulation of feature learning bias and amplifies the problems of label noise and class imbalance.

[0023] In the prior art, some long-tail noise label learning models ignore the impact of different noise rates in the data on model training, which will lead to deficiencies in high-noise environments because noise labels will destroy the reliability of samples and blur the distinction between noise samples and tail class samples, resulting in unsatisfactory performance under high-noise conditions; some long-tail noise label learning models only consider the class imbalance problem in long-tail learning, without considering the problem of a large number of noise labels in the data set, resulting in a sharp drop in model performance when both class imbalance and noise label problems exist.

[0024] When fine-tuning a large model, the present invention takes into account both the class imbalance problem and the noisy label problem. Different correction measures are taken by evaluating the noise rate of the data, with special attention paid to processing in high-noise scenarios. At the same time, the classification accuracy of the head and tail classes in high-noise scenarios is improved.

[0025] The following describes the model training method, device, equipment and medium for long-tail noise according to an embodiment of the present application with reference to the accompanying drawings. In view of the problem that the classification accuracy of head class and tail class samples in high noise scenarios in the related art mentioned in the above background technology is low, the present application provides a model training method for long-tail noise, in which an input image in a target long-tail noise data set, a text prompt word corresponding to the input image, and an observation label are input into a pre-trained visual language model, the visual language model includes a text encoder and an image encoder, and a fine-tuning module is provided in the image encoder; in the visual language model, the text prompt word is processed by the text encoder to obtain text features, the input image is processed by the image encoder to obtain image features, and the input image is processed by the image encoder with a fine-tuning module to obtain the original output value for each category; the text-image similarity between the text features and the image features is calculated, and a text prediction label is obtained based on the text-image similarity; the supervision start-stop state is determined based on the text prediction label and the observation label, the target loss function is determined based on the supervision start-stop state and the original output value, and the fine-tuning module is updated based on the target loss function to obtain a visual language model that has been trained. This application determines whether text-image alignment prior auxiliary supervision is needed by evaluating the difference between text prediction labels and observed labels, and improves the classification accuracy of head and tail class samples in high-noise scenarios by calibrating the deviation between learned features and observed labels.

[0026] See also Figure 1 The model training method for long-tail noise described in the embodiment of the present invention comprises the following steps: Step S100: input an input image in a target long-tail noise data set, text prompt words corresponding to the input image, and observation labels into a pre-trained visual language model, wherein the visual language model includes a text encoder and an image encoder, and the image encoder is provided with a fine-tuning module.

[0027] Specifically, the long-tail noise data set includes input images, text prompt words and observation labels, and the observation labels contain category information. Among them, one text prompt word corresponds to one observation label, and multiple images can correspond to the same text prompt word and observation label. The observation label may contain incorrect category information, while the text prompt word does not have the problem of incorrect labeling. Therefore, since the category information in the long-tail noise data set may be inconsistent with the corresponding image, the embodiment of the present application uses the text prompt words corresponding to the observation label to correct the inconsistency. The visual language model can be a CLIP model, which is a large text-image pre-training model using contrastive learning. The fine-tuning module is an adapter module added to the Transformer layer of the image encoder, which is used for fine-tuning of specific tasks and can be called an "adapter former".

[0028] That is, although the observed label does not match the input image, it still contains category information, namely text prompt words, and the present application uses text prompt words to correct the label image inconsistency problem in long-tail noise label data. Specifically, the present application uses the text encoder in the pre-trained visual language model to obtain text-based predictions, and uses this text-image alignment prior to correct the label image inconsistency. This supervisory signal, called weak teacher supervision (WTS), is not always accurate, so the present application evaluates the difference between the text prediction label and the observed label to decide when to activate weak teacher supervision. By calibrating the deviation between the learned features and labels, weak teacher supervision ultimately enhances the robustness of the model.

[0029] like Figure 1 As shown, the model training method for long-tail noise also includes the following steps: Step S200: In the visual language model, the text prompt word is processed by a text encoder to obtain text features, the input image is processed by an image encoder to obtain image features, and the input image is processed by the fine-tuning module to obtain original output values ​​for each category.

[0030] Specifically, the original output value (logits) is the linear output of the model for each category. It has not been converted to a probability distribution. These output values ​​can be regarded as the model's preliminary judgment or original output value for each category. The original output value is converted to a probability distribution only after the activation function (softmax). The text encoder of the visual language model can be a category The tag provides text embedding , the image encoder can be an input image Provide image embedding , by comparison and The similarity between them can be used to derive text prediction labels from the label-based text prompt words. The calculation formula for the text prediction label is: ; ; in, is the input image The text prediction label, i is the image number, T represents the text prompt word, Indicates a category. Indicates category The text features of the corresponding text prompt words, Represents the input image Image features and categories The text-image similarity between the text features of the corresponding text prompt words, Indicates the number of categories.

[0031] like Figure 1 As shown, the model training method for long-tail noise also includes the following steps: Step S300, calculating the text-image similarity between the text feature and the image feature, and obtaining a text prediction label based on the text-image similarity; Specifically, the text prediction label is also inaccurate. For example, its accuracy on the CIFAR-100 test set is only 64.4%, which shows that the first loss function introduces another form of label noise. To solve this problem, this application uses the text-image similarity between text and image to provide additional supervision. Because it can not only provide a discrete one-hot optimization target, but also provide information about the relationship between classes. Specifically, the activation function (softmax) is used to convert the text-image similarity into a probability, and the formula is: ; in, Indicates a category. Indicates that the input image belongs to the category The probability of Indicates Dimension, that is, categories, one category corresponds to one text prompt word, Represents the input image The image features and The text-image similarity between the text features of the dimensional text prompt words.

[0032] This application leverages the text encoder in a pre-trained visual language model (VLM) to obtain text-based predictions and uses this text-image alignment prior to correct for inconsistencies in labels and images.

[0033] like Figure 1 As shown, the model training method for long-tail noise also includes the following steps: Step S400, determining a supervision start / stop state based on the text prediction label and the observation label, determining a target loss function based on the supervision start / stop state and the original output value, and updating the fine-tuning module based on the target loss function to obtain a visual language model that has completed training.

[0034] However, using the text-image alignment prior as a supervisory signal is not always accurate. Therefore, the embodiment of the present application evaluates the difference between the text prediction label predicted by the text encoder and the observed label to decide whether to activate such supervision.

[0035] In an embodiment of the present application, determining the supervision start / stop state based on the text prediction label and the observation label includes: Calculating the overlap rate between the text prediction label and the observation label; Obtaining a preset control threshold, comparing the overlap rate with the control threshold, and obtaining a comparison result; The supervision start and stop status is determined according to the comparison result.

[0036] Specifically, since weak teachers are not always accurate and observation labels are more reliable in low-noise scenarios, observation labels can be used directly in low-noise scenarios without additional processing. However, since the proportion of noisy labels cannot be determined in advance, an indicator is needed to evaluate when the teacher provides effective supervision. The overlap ratio (OR) between text prediction labels and observation labels can be used as this indicator: ; in, Represents a text prediction label for a batch of data, Represents the observation labels of a batch of data.

[0037] This application determines the difference between the text prediction label and the observed label by calculating the overlap rate between the text prediction label and the observed label.

[0038] This application can eventually obtain a model that can effectively learn long-tail noisy label data and has high classification accuracy in each category by training on the target long-tail noise dataset. In other words, even if the training set has long-tail distribution and noisy labels, a model with high classification accuracy can still be trained.

[0039] In an embodiment of the present application, determining the supervision start / stop state according to the comparison result includes: If the comparison result is that the overlap rate is greater than the control threshold, the supervision start-stop state is a supervision stop state; If the comparison result is that the overlap rate is less than or equal to the control threshold, the supervision start-stop state is a supervision enabled state.

[0040] Specifically, if the text prediction label predicted by the pre-trained text encoder deviates greatly from the observed label, it means that the text prediction label contains rich and relatively accurate information. Therefore, the text prediction label is used to guide model training. Existing long-tail learning methods can effectively apply the method of the present application. Since text-based labels are usually less accurate than directly fine-tuning the image encoder, the text encoder is regarded as a "weak teacher" and the method of the present application is called weak teacher supervision (WTS). Multiple experiments conducted on benchmark datasets with various types of noisy labels and inherent long-tail distributions show that the weak teacher supervision proposed in this application improves the performance of strong students, especially in high noise ratio scenarios. The overall structural block diagram of weak teacher supervision is shown in the figure. Figure 2 shown.

[0041] When the overlap rate OR is high, it indicates that the text prediction label and the observed label are basically consistent, and the weak teacher supervision provides limited information. Therefore, weak teacher supervision is chosen to be disabled. On the contrary, when the OR is low, the observed label and the text prediction label (visual language alignment prior) are significantly different, indicating that auxiliary supervision is needed.

[0042] In an embodiment of the present application, determining a target loss function based on the supervision start / stop state and the original output value includes: If the supervision start-stop state is the supervision enabled state, calculating information divergence based on the text image similarity and the original output value, and using the information divergence as a first loss function; Obtain an image prediction label based on the original output value, and calculate a second loss function according to the image prediction label and the observation label; A target loss function is obtained based on the first loss function and the second loss function.

[0043] Specifically, the text image similarity is converted into a probability distribution, the original output value is converted into a probability distribution, and the text supervision information from the pre-trained visual language model is incorporated into the training process by minimizing the divergence between the probability distributions of the image and text prediction. This application adopts the Kullback-Leibler divergence (KL), that is, the first loss function is: ; in, is the probability distribution of all categories predicted by the text encoder, Represents the category predicted by the text encoder The probability distribution of is the probability distribution predicted by the image encoder with the fine-tuning module, represents the class predicted by the image encoder with the fine-tuning module Since the fine-tuning module has fewer training parameters and has the ability to quickly adapt to new datasets, this application regards the image encoder fine-tuned on long-tail noisy label data as a strong student, and the pre-trained visual language model as a weak teacher to provide weak teacher supervision (WTS).

[0044] The second loss function may adopt a cross entropy (CE) loss or an existing long-tail learning logarithmic adjustment method, such as an LDAM method and a LA method.

[0045] In the high-noise ratio scenario of the embodiment of the present application, weak teacher supervision shifts the focus from distinguishing noisy samples to solving the feature misalignment problem caused by noisy labels. This strategy helps to alleviate potential classification errors that may seriously affect the tail categories and improve the performance of all categories, including head and tail categories.

[0046] In one embodiment of the present application, obtaining a target loss function based on the first loss function and the second loss function includes: Setting a first weight for the first loss function and a second weight for the second loss function, wherein the sum of the first weight and the second weight is 1; The first loss function and the second loss function are weightedly summed based on the first weight and the second weight to obtain a target loss function.

[0047] Specifically, the calculation formula of the objective loss function is: ; in, is a hyperparameter, is the second loss function, For the input image The objective loss function For fine-tuning pre-trained vision-language models.

[0048] In an embodiment of the present application, determining a target loss function based on the supervision start / stop state and the original output value further includes: If the supervision start-stop state is a supervision disabled state, obtaining an image prediction label based on the original output value; A second loss function is calculated according to the image prediction label and the observation label, and the second loss function is used as the target loss function.

[0049] The OR-based supervision switch can be calculated as: ; in, is the control threshold of the coincidence rate, Represents a random variable The distribution parameter of and The embodiment of the present application estimates this value online by calculating the overlap rate of text prediction labels and observation labels in small batches of data, that is, the control threshold is obtained by calculating the overlap rate of text prediction labels and observation labels in preset batches of data. When weak teacher supervision needs to be turned off, , only the second loss function is included in the target loss function. In contrast, when weak teacher supervision is turned on, Set to a random number sampled from a beta distribution, which is a continuous probability distribution defined on the interval [0,1] and is often used to describe the probability of a random variable.

[0050] In the embodiment of the present application, in a low-noise scenario, weak teacher supervision is automatically disabled, which may introduce erroneous supervision, and only relies on observation labels that are considered to be relatively more reliable.

[0051] In one embodiment of the present application, obtaining an image prediction label based on the original output value includes: The original output value is converted into a probability distribution of each category using an activation function, and an image prediction label is obtained according to the probability distribution.

[0052] The present application obtains the probability of the input image from the similarity between the fine-tuned image features and the classifier weights, and then obtains the probability distribution of each category.

[0053] The weak teacher supervision (WTS) of this application has the following advantages: (1) Distribution bias-robust feature calibration. In high-noise ratio scenarios, weak teacher supervision shifts the focus from distinguishing noisy samples to solving the feature misalignment problem caused by noisy labels. This strategy helps to mitigate potential classification errors that may severely affect tail categories and improves the performance of all categories, including head and tail categories. In low-noise scenarios, weak teacher supervision automatically deactivates supervision that may introduce errors and relies only on observation labels that are considered relatively more reliable.

[0054] (2) Feature calibration that is robust to noisy labels. The pre-trained visual-language model is not affected by noisy labels and is aligned with the visual-language features, so it serves as a teacher model to correct the bias in the features obtained by the student model. It is worth noting that this application observes that fine-tuning using the second loss function can sometimes outperform text prediction, which is usually less accurate. Nevertheless, weak teacher supervision still provides valuable guidance in correcting the label-image mismatch introduced by noisy labels.

[0055] (3) Efficient training. The computational overhead introduced by weak teacher supervision is minimal, and the main computational overhead comes from the fine-tuning of the image encoder.

[0056] Table 1 shows the object classification accuracy (%) on the CIFAR-10-LTN and CIFAR-100-LTN datasets (joint noise).

[0057] Table 1

[0058] Table 2 shows the object classification accuracy (%) on the CIFAR-10-LTN and CIFAR-100-LTN datasets (symmetric noise, asymmetric noise), with an imbalance rate of 10.

[0059] Table 2

[0060] Table 3 shows the classification accuracy (%) on the Webvision-50 dataset (real noise).

[0061] Table 3

[0062] Table 4 shows the classification accuracy (%) on the Red mini-ImageNet dataset (real noise).

[0063] Table 4

[0064] Among them, CE (Cross Entropy) in the table refers to the cross entropy loss function, which is used for classification tasks to measure the difference between the predicted distribution and the true distribution. LDAM (Label-Distribution-Aware Margin Loss) is a margin adjustment loss function that considers label distribution and aims to improve long-tail learning. LDAM-DRW (Label Distribution Aware Margin with Deferred Re-Weighting) refers to the combination of LDAM and delayed reweighting, which addresses the long-tail distribution problem by adjusting the weights of different categories. NCM (Nearest Class Mean) refers to the nearest class mean classifier, which classifies by calculating the mean of each class and is suitable for processing unbalanced data. MiSLAS (Mixup Shifted Label-Aware Smoothing model) is a label smoothing learning method that enhances classifier learning and calibration by dealing with different degrees of class overconfidence. Co-teaching is a collaborative training method in which two models share data during training but make predictions and updates independently to reduce the impact of noise labels on the model. CDR (Combating noisylabels with Different Update Rules) uses different update strategies for different types of model parameters to reduce the impact of noisy labels. Sel-CL+ (Selective-supervised Contrastive Learning) performs supervised contrastive learning by selecting paired samples with high confidence to obtain robust pre-trained representations. RoLT (Robust Long-Tailed Learning under Label Noise) establishes a new prototype noise detection method to overcome the limitations of the small loss strategy, thereby improving the model's ability to detect noisy labels under long-tail distributions. RoLT-DRW refers to the combination of RoLT and delayed reweighting, which further improves the performance of the model by emphasizing the tail categories. HAR-DRW (Heteroskedastic Adaptive Regularization with Deferred Re-Weighting) combines delayed reweighting to perform stronger regularization on data points with high uncertainty and low density. RCAL (RepresentationCALibration) enhances the robustness of the model by calibrating the image representation obtained by self-supervised contrastive learning.ECBS (Extracting Clean and Balanced Subset for noisy long-tailed classification) proposes a pseudo-label framework for extracting clean and class-balanced subsets from long-tailed noisy datasets. ECBS-Res32 is an ECBS model based on a 32-layer residual network; ECBS-Res18 is an ECBS model based on an 18-layer residual network. LA (LogitAdjustment) performs long-tail learning by adjusting the logits of different categories. IB (Influence-Balanced Loss) adaptively assigns different weights to samples according to their influence on the decision boundary. DivdeMix is ​​a framework for noisy label learning using semi-supervised learning techniques. UNICON (UNIform selection and CONtrastivelearning) unifies the selection mechanism to ensure class balance in the selected clean samples. MW-Net (Meta-Weight-Net) automatically learns an explicit loss weight function in a meta-learning manner. HAR (HeteroskedasticAdaptive Regularization) performs stronger regularization on data points with high uncertainty and low density. ULC (Uncertainty-aware Label Correction framework) combines uncertainty-aware class-specific noise modeling and uncertainty-aware learning for label noise. TABASCO (Two-Stage Bi-Dimensional Sample Selection) is a two-stage noise sample separation strategy using two separation metrics. CT (Co-teaching) is a collaborative training method in which two models share data during training but make predictions and updates independently of each other to reduce the impact of noisy labels on the models. MentorNet is a method for supervising the training of a base deep network (student network). ELR+ (Early-Learning Regularization) exploits early learning through regularization to prevent the model from fitting noisy samples; MoPro (Momentum Prototypes) is a simple contrastive learning method that enables online noisy label correction, out-of-distribution sample removal, and representation learning. NGC (Noisy Graph Cleaning) collects clean samples by exploiting the geometric structure of the data and the confidence of model predictions. RCAL+ is an improved version of RCAL.LADE (LAbel distribution DisEntangling) refers to directly decomposing the source label distribution during the training phase, so that the model can effectively adapt to any target label distribution.

[0065] Tables 1 to 4 compare the classification accuracy of the method of this application and the conventional method. Experimental results show that the weak teacher supervision method proposed in this application can significantly improve the classification accuracy of each data set on the CLIP fine-tuning benchmark. All data sets that require artificial injection of noise labels adopt the strategy of first sampling into a long-tail distribution and then injecting the required noise labels according to the noise rate. This application conducted experiments on loss functions such as CE loss, LA loss, and LDAM loss, and used conventional joint noise, symmetric noise, asymmetric noise, and real noise as noise label types. Sufficient experiments have demonstrated the effectiveness of the method of this application.

[0066] In one embodiment, if Figure 3 As shown, based on the above-mentioned model training method for long-tail noise, the present invention also provides a model training device for long-tail noise, including: An image input module 100 is used to input an input image in a target long-tail noise dataset, a text prompt word corresponding to the input image, and an observation label into a pre-trained visual language model, wherein the visual language model includes a text encoder and an image encoder, and the image encoder is provided with a fine-tuning module; A feature extraction module 200 is used for obtaining text features after the text prompt word is processed by a text encoder, obtaining image features after the input image is processed by an image encoder, and obtaining original output values ​​for each category after the input image is processed by the image encoder with a fine-tuning module in the visual language model; A text prediction module 300, configured to calculate the text-image similarity between the text feature and the image feature, and obtain a text prediction label based on the text-image similarity; The updating module 400 is used to determine the supervision start and stop state based on the text prediction label and the observation label, determine the target loss function based on the supervision start and stop state and the original output value, and update the fine-tuning module based on the target loss function to obtain a visual language model that has completed training.

[0067] It should be noted that the aforementioned explanation of the embodiment of the model training method for long-tail noise is also applicable to the model training device for long-tail noise in this embodiment, and will not be repeated here.

[0068] The invention discloses a model training device for long-tail noise. The device comprises an input image in a target long-tail noise data set, a text prompt word corresponding to the input image and an observation label, which are input into a pre-trained visual language model. The visual language model comprises a text encoder and an image encoder, and a fine-tuning module is arranged in the image encoder. In the visual language model, the text prompt word is processed by the text encoder to obtain text features, the input image is processed by the image encoder to obtain image features, and the input image is processed by the image encoder with the fine-tuning module to obtain original output values ​​for each category. The text-image similarity between the text features and the image features is calculated, and a text prediction label is obtained based on the text-image similarity. The supervision start-stop state is determined based on the text prediction label and the observation label, a target loss function is determined based on the supervision start-stop state and the original output value, and the fine-tuning module is updated based on the target loss function to obtain a visual language model that has been trained. This application determines whether text-image alignment prior auxiliary supervision is needed by evaluating the difference between text prediction labels and observed labels, and improves the classification accuracy of head and tail class samples in high-noise scenarios by calibrating the deviation between learned features and observed labels.

[0069] Figure 4 A schematic diagram of the structure of the device provided in the embodiment of the present application. The device may include: A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0070] When the processor 502 executes the program, the model training method for long-tail noise provided in the above embodiment is implemented.

[0071] Furthermore, the device also includes: The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0072] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0073] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0074] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0075] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0076] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0077] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the model training method for long-tail noise is implemented as described above.

[0078] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0079] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0080] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, in which the order shown or discussed may not be followed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0081] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can read instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary and then storing it in a computer memory.

[0082] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0083] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0084] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0085] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A model training method for long-tail noise, characterized in that: The method comprises: Inputting an input image in a target long-tail noise dataset, a text prompt word corresponding to the input image, and an observation label into a pre-trained visual language model, wherein the visual language model includes a text encoder and an image encoder, and the image encoder is provided with a fine-tuning module; In the visual language model, the text prompt word is processed by a text encoder to obtain text features, the input image is processed by an image encoder to obtain image features, and the input image is processed by the image encoder with a fine-tuning module to obtain original output values ​​for each category; Calculating the text-image similarity between the text feature and the image feature, and obtaining a text prediction label based on the text-image similarity; The supervision start and stop state is determined based on the text prediction label and the observation label, the target loss function is determined based on the supervision start and stop state and the original output value, and the fine-tuning module is updated based on the target loss function to obtain a visual language model that has completed training.

2. The model training method for long-tail noise according to claim 1, characterized in that: Determining the supervision start / stop state based on the text prediction label and the observation label includes: Calculating the overlap rate between the text prediction label and the observation label; Obtaining a preset control threshold, comparing the overlap rate with the control threshold, and obtaining a comparison result; The supervision start and stop status is determined according to the comparison result.

3. The model training method for long-tail noise according to claim 2, characterized in that: Determining the supervision start and stop state according to the comparison result includes: If the comparison result is that the overlap rate is greater than the control threshold, the supervision start-stop state is a supervision stop state; If the comparison result is that the overlap rate is less than or equal to the control threshold, the supervision start-stop state is a supervision enabled state.

4. The model training method for long-tail noise according to claim 3, characterized in that: Determining a target loss function based on the supervised start / stop state and the original output value includes: If the supervision start-stop state is the supervision enabled state, calculating information divergence based on the text image similarity and the original output value, and using the information divergence as a first loss function; Obtain an image prediction label based on the original output value, and calculate a second loss function according to the image prediction label and the observation label; A target loss function is obtained based on the first loss function and the second loss function.

5. The model training method for long-tail noise according to claim 4, characterized in that: Obtaining a target loss function based on the first loss function and the second loss function includes: Setting a first weight for the first loss function and a second weight for the second loss function, wherein the sum of the first weight and the second weight is 1; The first loss function and the second loss function are weightedly summed based on the first weight and the second weight to obtain a target loss function.

6. The model training method for long-tail noise according to claim 3, characterized in that: Determining a target loss function based on the supervised start / stop state and the original output value further includes: If the supervision start-stop state is a supervision disabled state, obtaining an image prediction label based on the original output value; A second loss function is calculated according to the image prediction label and the observation label, and the second loss function is used as the target loss function.

7. The model training method for long-tail noise according to claim 4 or 6, characterized in that: The image prediction label is obtained based on the original output value, including: The original output value is converted into a probability distribution of each category using an activation function, and an image prediction label is obtained according to the probability distribution.

8. A model training device for long-tail noise, characterized in that: The device comprises: An image input module is used to input an input image in a target long-tail noise dataset, a text prompt word corresponding to the input image, and an observation label into a pre-trained visual language model, wherein the visual language model includes a text encoder and an image encoder, and the image encoder is provided with a fine-tuning module; A feature extraction module, used for obtaining text features after the text prompt word is processed by a text encoder, obtaining image features after the input image is processed by an image encoder, and obtaining original output values ​​for each category after the input image is processed by the image encoder with a fine-tuning module in the visual language model; A text prediction module, used to calculate the text-image similarity between the text feature and the image feature, and obtain a text prediction label based on the text-image similarity; An updating module is used to determine a supervision start / stop state based on the text prediction label and the observation label, determine a target loss function based on the supervision start / stop state and the original output value, and update the fine-tuning module based on the target loss function to obtain a visual language model that has completed training.

9. A device, characterized in that: include: A memory, a processor, and a model training program for long-tail noise stored in the memory and executable on the processor, wherein the model training program for long-tail noise, when executed by the processor, implements the steps of the model training method for long-tail noise as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed to implement the steps of the model training method for long-tail noise as described in any one of claims 1 to 7.

Images