Image classification method based on visual language model noise label optimization

Abstract

The invention discloses an image classification method based on visual language model noise tag optimization, which comprises the following steps: acquiring a to-be-processed image and inputting the to-be-processed image into an image classification model to obtain a classification result; wherein the image classification model is obtained based on high-quality sample set training. In the process of screening high-quality samples, in the first stage, the loss value of each sample image is calculated by utilizing a CLIP model in combination with macroscopic and microscopic text prompts, and the loss value of each sample image is fitted through a two-component Gaussian mixture model, so that the upper bound and the lower bound of a confusion sample are determined according to two distributions; dividing to obtain clean samples, confused samples and noise samples; and in the second stage, different strategies are adopted to determine pseudo labels of the three samples, each sample image and the pseudo label thereof are input into the BLIP model, and the similarity score of the sample image and the pseudo label thereof is calculated so as to screen out high-quality samples. According to the method, the self-confirmation deviation of the image classification model can be reduced while the label quality is ensured, so that the image classification model has good robustness.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to an image classification method based on noise label optimization of a visual language model. Background Art

[0002] In the fields of machine learning and deep learning, noisy labels, caused by inaccurate data annotation and inherent data complexity, are a common problem during training, severely impacting the performance of neural network models. Therefore, it is crucial to understand how to ensure the performance of neural network models in the presence of noise.

[0003] Existing technologies use methods such as small loss mechanisms and self-supervision constraints to deal with noisy labels. However, these methods still fail to effectively solve the performance degradation problem caused by noisy labels due to problems such as self-confirmation bias and the limitations of traditional small loss mechanisms.

[0004] In recent years, visual language models have demonstrated great potential in downstream tasks. Consequently, some research has explored how to fine-tune these models to address the noisy labeling problem. For example, Guo et al. used the pre-trained visual language model CLIP as a baseline network to design an adaptive thresholding algorithm to address the noisy labeling problem. However, this approach still fails to address self-confirmation bias when faced with higher or more complex noise levels, potentially impacting model performance due to noisy labels. Summary of the Invention

[0005] To address the above-mentioned problems in the prior art, the present invention provides an image classification method based on noise label optimization using a visual language model. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides an image classification method based on noise label optimization of a visual language model, comprising:

[0007] Get the image to be processed;

[0008] Input the image to be processed into an image classification model to obtain a classification result of the image to be processed; wherein the image classification model is trained based on multiple high-quality samples in a high-quality sample set, and the high-quality samples are screened according to the following steps:

[0009] Acquire a plurality of training samples, the plurality of training samples comprising: a plurality of first training samples consisting of the i-th sample image and macro text prompts of various categories, and a plurality of second training samples consisting of the i-th sample image and micro text prompts of various categories;

[0010] Inputting a plurality of first training samples including the i-th sample image into the contrastive language-image pre-trained CLIP model to obtain a prediction result of the i-th sample image when the text input is a macro text prompt, and inputting a plurality of second training samples including the i-th sample image into the contrastive language-image pre-trained CLIP model to obtain a prediction result of the i-th sample image when the text input is a micro text prompt;

[0011] According to the prediction result of the i-th sample image when the text input is a macro text prompt, the prediction result of the i-th sample image when the text input is a micro text prompt, and a preset loss function, the loss value corresponding to each sample image is calculated to classify the sample image into a clean sample, a confused sample, or a noisy sample based on the loss value;

[0012] Using different strategies to determine pseudo labels for the clean sample, the obfuscated sample, and the noise sample;

[0013] Use the pre-trained BLIP model of Unified Visual Language Understanding and Generation to calculate the similarity score between each clean sample and its pseudo-label, the similarity score between each confused sample and its pseudo-label, and the similarity score between each noisy sample and its pseudo-label;

[0014] Based on the similarity scores, high-quality sample images and their pseudo labels are selected to form a high-quality sample set.

[0015] In one embodiment of the present invention, the comparative language-image pre-trained CLIP model includes a text encoder and a visual encoder;

[0016] The step of inputting a plurality of first training samples including the i-th sample image into a language-image pre-trained CLIP model for comparison, and obtaining a prediction result of the i-th sample image when the text input is a macro text prompt, comprises:

[0017] For a plurality of first training samples including the i-th sample image, the i-th sample image is input into the visual encoder and the macro text prompt of category j is input into the text encoder to obtain the image feature vector f i and macro text embedding vectors C represents the total number of categories;

[0018] According to the image feature vector f i and macro text embedding vectors Calculate the probability that the i-th sample image belongs to category j when the text input is a macro text prompt τ is the temperature coefficient.

[0019] In one embodiment of the present invention, the step of inputting a plurality of second training samples including the i-th sample image into the comparative language-image pre-training CLIP model to obtain a prediction result of the i-th sample image when the text input is a micro text prompt includes:

[0020] For multiple second training samples including the i-th sample image, the i-th sample image is input into the visual encoder and the micro text prompt of category j is input into the text encoder to obtain the image feature vector f i and micro-text embedding vectors

[0021] According to the image feature vector f i and the micro text embedding vector Calculate the probability that the i-th sample image belongs to category j when the text input is a micro text prompt

[0022] In one embodiment of the present invention, the step of calculating a loss value corresponding to each sample image based on a prediction result of the i-th sample image when the text input is a macro text prompt, a prediction result of the i-th sample image when the text input is a micro text prompt, and a preset loss function, and classifying the sample image into a clean sample, a confused sample, or a noise sample based on the loss value includes:

[0023] The probability that the i-th sample image belongs to category j when the text input is a macro text prompt The probability that the i-th sample image belongs to category j when the text input is a micro text prompt And the preset loss function is used to calculate the loss value of the i-th sample image;

[0024] Use a two-component Gaussian mixture model to fit the loss values of all sample images to determine the upper and lower bounds of the confused samples;

[0025] Based on the upper bound and the lower bound, a three-stage screening strategy is used to divide all sample images into clean samples, confused samples and noise samples.

[0026] In one embodiment of the present invention, the loss value of the i-th sample is:

[0027] l(x i ,y i )=l ce (x i ,y i )+λ·l con (x i )+β·l ent (x i );

[0028] In the formula, the cross entropy loss value is Consistency constraint loss value Entropy regularization loss value y i represents the observed label of the i-th sample image, λ and β are both preset weights. When the true category of the i-th sample image is category j, y i,j =1, when the true category of the i-th sample image is not category j, y i,j =0.

[0029] In one embodiment of the present invention, the step of fitting the loss values of all sample images using a two-component Gaussian mixture model to determine the upper and lower bounds of the confused samples includes:

[0030] Use a two-component Gaussian mixture model to fit the loss values of all sample images:

[0031]

[0032] Where l represents the loss value, (μ1,σ1) represents the distribution f c The mean and standard deviation of (μ2,σ2) represent the distribution f n The mean and standard deviation of

[0033] Based on (μ1,σ1), (μ2,σ2) and the first preset threshold θ, the boundary of the confused sample is calculated:

[0034]

[0035]

[0036] The larger of α1 and α2 is used as the upper bound of the confused sample, and the smaller of α1 and α2 is used as the lower bound of the confused sample.

[0037] In one embodiment of the present invention, the step of dividing all sample images into clean samples, obfuscated samples, and noisy samples using a three-stage screening strategy based on the upper bound and the lower bound includes:

[0038] The sample images whose loss values are less than the lower bound are classified as clean samples, the sample images whose loss values are greater than or equal to the lower bound and less than the upper bound are classified as confused samples, and the sample images whose loss values are greater than or equal to the upper bound are classified as noise samples.

[0039] In one embodiment of the present invention, the step of using different strategies to determine the pseudo labels of the clean sample, the obfuscated sample, and the noise sample includes:

[0040] Take the observed labels of clean samples as their pseudo labels;

[0041] Get the predicted category of the noise sample by the contrastive language-image pre-trained CLIP model as its pseudo label;

[0042] Obtain the predicted category of the confused sample and the observed label of the confused sample by the contrastive language-image pre-trained CLIP model, and obtain its pseudo label after weighting; where,

[0043] The predicted category of the contrastive language-image pre-trained CLIP model for the i-th sample image is: the probability that the i-th sample image belongs to category j when the text input is a micro text prompt The probability that the i-th sample image belongs to category j when the text input is a micro text prompt After taking the average, the category corresponding to the maximum value of the average probability is obtained.

[0044] In one embodiment of the present invention, before the step of selecting high-quality sample images and their pseudo labels based on similarity scores to form a high-quality sample set, the following steps are further included:

[0045] The two-component Gaussian mixture model is used to fit the similarity scores of all samples, and two distributions are obtained:

[0046]

[0047]

[0048] Where s represents the similarity score, (μ3,σ3) represents the distribution f low The mean and standard deviation of (μ4,σ4) represent the distribution f high The mean and standard deviation of .

[0049] In one embodiment of the present invention, the step of selecting high-quality sample images and their pseudo labels based on similarity scores to form a high-quality sample set includes:

[0050] Calculate the distribution f with a larger mean for each sample image high the posterior probability of (s);

[0051] The sample images whose posterior probability is less than the second preset threshold are taken as low-quality sample images, and the sample images whose posterior probability is greater than or equal to the second preset threshold are taken as high-quality sample images, and pseudo labels of the high-quality sample images are further obtained to form a high-quality sample set.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] In the first stage, the present invention uses the CLIP model, combined with macro- and micro-textual cues, to calculate the loss value for each sample image. This loss value is then used as a screening criterion. A two-component Gaussian mixture model is then used to fit the loss value of each sample image. This determines the upper and lower bounds of obfuscated samples based on the two distributions, and classifies all sample images into clean, obfuscated, and noisy samples. In the second stage, different strategies are used to determine pseudo-labels for these three types of samples. Each sample and its pseudo-label are then fed into the BLIP model, where a similarity score is calculated to select high-quality samples.

[0054] The present invention utilizes a visual language model to screen high-quality samples and uses these high-quality samples to train an image classification model. This can ensure label quality while reducing the self-confirmation bias of the image classification model, making the image classification model have good robustness.

[0055] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of an image classification method based on visual language model noise label optimization provided by an embodiment of the present invention;

[0057] Figure 2 Schematic diagram of the process of dividing clean samples, noisy samples and confused samples provided by an embodiment of the present invention;

[0058] Figure 3 Schematic diagram of the process of screening high-quality samples provided by an embodiment of the present invention.

[0059] Figure 4 3 is a schematic diagram of the distribution of loss values of all sample images fitted by the two-component Gaussian mixture model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0061] Figure 1 This is a flow chart of an image classification method based on visual language model noise label optimization provided by an embodiment of the present invention. Figure 2 Schematic diagram of the process of dividing clean samples, noisy samples and confused samples provided by an embodiment of the present invention. Figure 3 This is a schematic diagram of the process of screening high-quality samples provided by an embodiment of the present invention. Figures 1 to 3 The embodiment of the present invention provides an image classification method based on noise label optimization of a visual language model, comprising:

[0062] S1, obtain the image to be processed;

[0063] S2. Input the image to be processed into an image classification model to obtain a classification result of the image to be processed; wherein the image classification model is trained based on multiple high-quality samples in the high-quality sample set, and the high-quality samples are screened according to the following steps:

[0064] S21. Acquire multiple training samples, where the multiple training samples include: multiple first training samples consisting of the i-th sample image and macro text prompts of various categories, and multiple second training samples consisting of the i-th sample image and micro text prompts of various categories.

[0065] S22. Input multiple first training samples including the i-th sample image into the CLIP model (Contrastive Language-Image Pre-training) to obtain a prediction result of the i-th sample image when the text input is a macro text prompt. Input multiple second training samples including the i-th sample image into the contrastive language-image pre-training CLIP model to obtain a prediction result of the i-th sample image when the text input is a micro text prompt.

[0066] Specifically, Figure 2 Taking the bombay, beagle, and birman categories shown in the figure as examples, the macro text prompts are "a photo of a bombay," "a photo of a beagle," and "a photo of a birman," respectively. Correspondingly, the micro text prompts are "a photo of a bambay, which is a muscular and compact cat," "a photo of a beagle, which is a friendly and curious breed," and "a photo of a birman, which has striking blue eyes." It should be noted that macro and micro text prompts can be obtained through manual annotation.

[0067] Furthermore, for each sample image, it needs to be combined with macro text prompts of the categories of bombay, beagle, and birman to form multiple first training samples, and combined with micro text prompts of these categories to form multiple second training samples.

[0068] Optionally, the comparative language-image pre-trained CLIP model includes a text encoder and a visual encoder;

[0069] In step S22, a plurality of first training samples including the i-th sample image are input into a comparative language-image pre-trained CLIP model to obtain a prediction result of the i-th sample image when the text input is a macro text prompt, including:

[0070] For multiple first training samples including the i-th sample image, the i-th sample image is input into the visual encoder and the macro text prompt of category j is input into the text encoder to obtain the image feature vector f i and macro text embedding vectors C represents the total number of categories;

[0071] According to the image feature vector f i and macro text embedding vectors Calculate the probability that the i-th sample image belongs to category j when the text input is a macro text prompt τ is the temperature coefficient.

[0072] Furthermore, the step of inputting a plurality of second training samples including the i-th sample image into the comparative language-image pre-training CLIP model to obtain a prediction result of the i-th sample image when the text input is a micro text prompt includes:

[0073] For multiple second training samples including the i-th sample image, the i-th sample image is input into the visual encoder and the micro text prompt of category j is input into the text encoder to obtain the image feature vector f i and micro-text embedding vectors

[0074] According to the image feature vector f i and micro-text embedding vectors Calculate the probability that the i-th sample image belongs to category j when the text input is a micro text prompt

[0075] S23. Calculate the loss value corresponding to each sample image based on the prediction result of the i-th sample image when the text input is a macro text prompt, the prediction result of the i-th sample image when the text input is a micro text prompt, and a preset loss function, so as to divide the sample image into a clean sample, a confused sample, or a noise sample based on the loss value.

[0076] Specifically, step S23 includes:

[0077] S231, the probability that the i-th sample image belongs to category j when the text input is a macro text prompt The probability that the i-th sample image belongs to category j when the text input is a micro text prompt And the preset loss function is used to calculate the loss value of the i-th sample image.

[0078] In this embodiment, the loss value of the i-th sample is:

[0079] l(x i ,y i )=l ce (x i ,y i )+λ·l con (x i )+β·l ent (x i );

[0080] In the formula, the cross entropy loss value is Consistency constraint loss value Entropy regularization loss value y i represents the observed label of the i-th sample image, λ and β are both preset weights. When the true category of the i-th sample image is category j, y i,j =1, when the true category of the i-th sample image is not category j, y i,j =0.

[0081] It should be noted that the observed labels are not necessarily accurate. For example, for a sample image of a dog, the true label is "dog", but because the sample image itself contains noise, the observed label may be "cat". Generally, the observed labels can be manually annotated.

[0082] S232. Use a two-component Gaussian mixture model to fit the loss values of all sample images to determine the upper and lower bounds of the confused samples.

[0083] Figure 4 This is a schematic diagram of the distribution of loss values of all sample images fitted by the two-component Gaussian mixture model provided by the embodiment of the present invention. Figure 4 , first, use a two-component Gaussian mixture model to fit the loss values of all sample images:

[0084]

[0085] Where l represents the loss value, (μ1,σ1) represents the distribution f cThe mean and standard deviation of (μ2,σ2) represent the distribution f n The mean and standard deviation of

[0086] Next, based on (μ1, σ1), (μ2, σ2) and the first preset threshold θ, the boundary of the confused sample is calculated:

[0087]

[0088] The larger of α1 and α2 is used as the upper bound of the confused sample, and the smaller of α1 and α2 is used as the lower bound of the confused sample.

[0089] S233. Based on the upper bound and the lower bound, a three-stage screening strategy is used to divide all sample images into clean samples, confused samples and noise samples.

[0090] In this step, sample images with loss values less than the lower bound are classified as clean samples, sample images with loss values greater than or equal to the lower bound and less than the upper bound are classified as confused samples, and sample images with loss values greater than or equal to the upper bound are classified as noise samples.

[0091] S24. Use different strategies to determine the pseudo labels of clean samples, confused samples and noisy samples.

[0092] Specifically, the observed labels of clean samples are used as their pseudo labels;

[0093] Get the predicted category of the noise sample by the contrastive language-image pre-trained CLIP model as its pseudo label;

[0094] Obtain the predicted category of the confused sample and the observed label of the confused sample by the contrastive language-image pre-trained CLIP model, and obtain its pseudo label after weighting; where,

[0095] It should be understood that the predicted category of the i-th sample image by the contrastive language-image pre-trained CLIP model is: the probability that the i-th sample image belongs to category j when the text input is a micro text prompt The probability that the i-th sample image belongs to category j when the text input is a micro text prompt After taking the average, the category corresponding to the maximum value of the average probability is obtained.

[0096] S25. Use the BLIP model (Bootstrapping Language-Image Pre-training, a pre-training model for unified visual language understanding and generation) to calculate the similarity score between each clean sample and its pseudo-label, the similarity score between each confused sample and its pseudo-label, and the similarity score between each noise sample and its pseudo-label.

[0097] Optionally, before the step of selecting high-quality sample images and their pseudo labels based on the similarity scores to form a high-quality sample set, the method further includes:

[0098] The two-component Gaussian mixture model is used to fit the similarity scores of all samples, and two distributions are obtained:

[0099]

[0100] Where s represents the similarity score, (μ3,σ3) represents the distribution f low The mean and standard deviation of (μ4,σ4) represent the distribution f high The mean and standard deviation of .

[0101] S26. Select high-quality sample images and their pseudo labels based on the similarity scores to form a high-quality sample set.

[0102] In step S26, the distribution f with the largest mean value is calculated for each sample image. high the posterior probability of (s);

[0103] The sample images whose posterior probability is less than the second preset threshold are regarded as low-quality sample images, and the sample images whose posterior probability is greater than or equal to the second preset threshold are regarded as high-quality sample images, and pseudo labels of the high-quality sample images are further obtained to form a high-quality sample set.

[0104] Next, the image classification method based on visual language model noise label optimization provided by the present invention is further illustrated through simulation experiments.

[0105] This simulation was conducted on an Intel XEON E5-2680V4 209 CPU, an NVIDIA GTX 4090 GPU, and the Ubuntu 16.04 operating system, using PyTorch 1.13.1, an open-source library from Facebook. The datasets used included 10 datasets: Flowers102, EuroSAT, StanfordCars, OxfordPets, DTD, Caltech101, UCF01, Food101, ImageNet, and SUN397. These datasets cover a range of visual tasks, including classification of general objects, scenes, actions, and fine-grained categories, as well as texture recognition and satellite imagery.

[0106] The database uses two different types of noise, including symflip noise and pairflip noise. The noise label generation method is the existing technology and will not be explained here.

[0107] This simulation compared the noise label processing method used by our invention with the existing JoAPR method. Specifically, the JoAPR method introduces an adaptive threshold, dynamically updating the threshold for selecting clean and noisy samples based on the noise rate at different times. As shown in Table 1, after training on 10 databases, our invention demonstrated higher image classification accuracy, achieving optimal results in all noise conditions.

[0108] Table 1

[0109]

[0110]

[0111] It can be seen that this method uses the visual language model to optimize the labels of training samples, and uses the optimized high-quality sample set to train the image classification model, which can enable the image classification model to maintain good robustness under various noise conditions.

[0112] It can be seen from the above embodiments that the beneficial effects of the present invention are:

[0113] In the first stage, the present invention uses the CLIP model, combined with macro- and micro-textual cues, to calculate the loss value for each sample image. This loss value is then used as a screening criterion. A two-component Gaussian mixture model is then used to fit the loss value of each sample image. This determines the upper and lower bounds of obfuscated samples based on the two distributions, and classifies all sample images into clean, obfuscated, and noisy samples. In the second stage, different strategies are used to determine pseudo-labels for these three types of samples. Each sample and its pseudo-label are then fed into the BLIP model, where a similarity score is calculated to select high-quality samples.

[0114] The present invention utilizes a visual language model to screen high-quality samples and uses these high-quality samples to train an image classification model. This can ensure label quality while reducing the self-confirmation bias of the image classification model, making the image classification model have good robustness.

[0115] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0116] Descriptions with reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0117] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. An image classification method based on noise label optimization of visual language model, characterized in that: include: Get the image to be processed; Input the image to be processed into an image classification model to obtain a classification result of the image to be processed; wherein the image classification model is trained based on multiple high-quality samples in a high-quality sample set, and the high-quality samples are screened according to the following steps: Acquire a plurality of training samples, the plurality of training samples comprising: a plurality of first training samples consisting of the i-th sample image and macro text prompts of various categories, and a plurality of second training samples consisting of the i-th sample image and micro text prompts of various categories; Inputting a plurality of first training samples including the i-th sample image into the contrastive language-image pre-trained CLIP model to obtain a prediction result of the i-th sample image when the text input is a macro text prompt, and inputting a plurality of second training samples including the i-th sample image into the contrastive language-image pre-trained CLIP model to obtain a prediction result of the i-th sample image when the text input is a micro text prompt; According to the prediction result of the i-th sample image when the text input is a macro text prompt, the prediction result of the i-th sample image when the text input is a micro text prompt, and a preset loss function, the loss value corresponding to each sample image is calculated to classify the sample image into a clean sample, a confused sample, or a noisy sample based on the loss value; Using different strategies to determine pseudo labels for the clean sample, the obfuscated sample, and the noise sample; Use the pre-trained BLIP model of Unified Visual Language Understanding and Generation to calculate the similarity score between each clean sample and its pseudo-label, the similarity score between each confused sample and its pseudo-label, and the similarity score between each noisy sample and its pseudo-label; Based on the similarity scores, high-quality sample images and their pseudo labels are selected to form a high-quality sample set.

2. The image classification method based on visual language model noise label optimization according to claim 1 is characterized in that The comparative language-image pre-training CLIP model includes a text encoder and a visual encoder; The step of inputting a plurality of first training samples including the i-th sample image into a language-image pre-trained CLIP model for comparison, and obtaining a prediction result of the i-th sample image when the text input is a macro text prompt, comprises: For a plurality of first training samples including the i-th sample image, the i-th sample image is input into the visual encoder and the macro text prompt of category j is input into the text encoder to obtain the image feature vector f i and macro text embedding vectors j=1,2,…,C, where C represents the total number of categories; According to the image feature vector f i and macro text embedding vectors Calculate the probability that the i-th sample image belongs to category j when the text input is a macro text prompt τ is the temperature coefficient.

3. The image classification method based on visual language model noise label optimization according to claim 2 is characterized in that The step of inputting a plurality of second training samples including the i-th sample image into the comparative language-image pre-training CLIP model to obtain a prediction result of the i-th sample image when the text input is a micro text prompt comprises: For multiple second training samples including the i-th sample image, the i-th sample image is input into the visual encoder and the micro text prompt of category j is input into the text encoder to obtain the image feature vector f i and micro-text embedding vectors According to the image feature vector f i and the micro text embedding vector Calculate the probability that the i-th sample image belongs to category j when the text input is a micro text prompt 4. The image classification method based on visual language model noise label optimization according to claim 3 is characterized in that The step of calculating a loss value corresponding to each sample image based on a prediction result of the i-th sample image when the text input is a macro text prompt, a prediction result of the i-th sample image when the text input is a micro text prompt, and a preset loss function, and classifying the sample image into a clean sample, a confused sample, or a noise sample based on the loss value, includes: The probability that the i-th sample image belongs to category j when the text input is a macro text prompt The probability that the i-th sample image belongs to category j when the text input is a micro text prompt And the preset loss function is used to calculate the loss value of the i-th sample image; Use a two-component Gaussian mixture model to fit the loss values of all sample images to determine the upper and lower bounds of the confused samples; Based on the upper bound and the lower bound, all sample images are divided into clean samples, confused samples and noise samples using a three-stage screening strategy.

5. The image classification method based on visual language model noise label optimization according to claim 4 is characterized in that: The loss value of the i-th sample is: l(x i ,y i )=l ce (x i ,y i )+λ·l con (x i )+β·l ent (x i ); In the formula, the cross entropy loss value is Consistency constraint loss value Entropy regularization loss value y i represents the observed label of the i-th sample image, λ and β are both preset weights. When the true category of the i-th sample image is category j, y i,j =1, when the true category of the i-th sample image is not category j, y i,j =0.

6. The image classification method based on visual language model noise label optimization according to claim 4 is characterized in that The steps of fitting the loss values of all sample images using a two-component Gaussian mixture model to determine the upper and lower bounds of the confused samples include: Use a two-component Gaussian mixture model to fit the loss values of all sample images: Where l represents the loss value, (μ1,σ1) represents the distribution f c The mean and standard deviation of (μ2,σ2) represent the distribution f n The mean and standard deviation of Based on (μ1,σ1), (μ2,σ2) and the first preset threshold θ, the boundary of the confused sample is calculated: The larger of α1 and α2 is used as the upper bound of the confused sample, and the smaller of α1 and α2 is used as the lower bound of the confused sample.

7. The image classification method based on visual language model noise label optimization according to claim 6, characterized in that: Based on the upper bound and the lower bound, the steps of using a three-stage screening strategy to divide all sample images into clean samples, confused samples, and noisy samples include: The sample images whose loss values are less than the lower bound are classified as clean samples, the sample images whose loss values are greater than or equal to the lower bound and less than the upper bound are classified as confused samples, and the sample images whose loss values are greater than or equal to the upper bound are classified as noise samples.

8. The image classification method based on visual language model noise label optimization according to claim 1, characterized in that: The steps of determining pseudo labels of the clean sample, the obfuscated sample, and the noise sample using different strategies include: Take the observed labels of clean samples as their pseudo labels; Get the predicted category of the noise sample by the contrastive language-image pre-trained CLIP model as its pseudo label; Obtain the predicted category of the confused sample and the observed label of the confused sample by the contrastive language-image pre-trained CLIP model, and obtain its pseudo label after weighting; where, The predicted category of the contrastive language-image pre-trained CLIP model for the i-th sample image is: the probability that the i-th sample image belongs to category j when the text input is a micro text prompt The probability that the i-th sample image belongs to category j when the text input is a micro text prompt After taking the average, the category corresponding to the maximum value of the average probability is obtained.

9. The image classification method based on visual language model noise label optimization according to claim 7, characterized in that: Before selecting high-quality sample images and their pseudo labels based on the similarity scores to form a high-quality sample set, the following steps are also included: The two-component Gaussian mixture model is used to fit the similarity scores of all samples, and two distributions are obtained: Where s represents the similarity score, (μ3,σ3) represents the distribution f low The mean and standard deviation of (μ4,σ4) represent the distribution f high The mean and standard deviation of .

10. The image classification method based on visual language model noise label optimization according to claim 9, characterized in that: The steps of selecting high-quality sample images and their pseudo labels based on similarity scores to form a high-quality sample set include: Calculate the distribution f with a larger mean for each sample image high the posterior probability of (s); The sample images whose posterior probability is less than the second preset threshold are taken as low-quality sample images, and the sample images whose posterior probability is greater than or equal to the second preset threshold are taken as high-quality sample images, and pseudo labels of the high-quality sample images are further obtained to form a high-quality sample set.

Images