Unsupervised image classification training method and system based on visual language model, and medium

Abstract

The invention discloses an unsupervised image classification training method and system based on a visual language model and a medium, and relates to the technical field of computers, and the method comprises the following steps: building a training data set D without manually annotated classification labels and a test set with manually annotated classification labels by experts; performing zero sample reasoning on the training data set based on a visual language model to obtain a pseudo tag; screening the pseudo labels by using a multi-view consistency strategy to obtain a first training subset Dil; screening by using a classification-clustering result consistency strategy to obtain a second training subset D1; a high-confidence cross-supervised learning method is adopted, and the second training subset D1 and the remaining unlabeled training subsets Du are utilized to train an image classification model; the method has the advantages that manual labeling is not needed, high-accuracy classification is achieved, the negative influence of noise labels on model training is reduced, the model is trained in a random mixing mode, and the generalization ability of the model can be enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an unsupervised image classification training method, system and medium based on a visual language model. Background Art

[0002] Image classification is a technology that automatically categorizes images into specific categories. It is widely used in various fields, including industry, healthcare, and security, and can improve image processing speed and the level of automated decision-making. Image classification technologies are mainly divided into two categories: feature engineering-based and deep learning-based. Feature engineering-based methods rely primarily on manually designed feature extraction and traditional machine learning methods. Their disadvantages include limited feature extraction capabilities for complex images, sensitivity to noise, the need for extensive manual intervention for feature selection and parameter adjustment, and difficulty scalability to large datasets. Deep learning methods automatically extract image features by learning from large amounts of data, are highly adaptable, and offer higher classification accuracy.

[0003] Labeled data is crucial for training accurate deep learning models, but labeling large amounts of data for specific tasks is extremely time-consuming and labor-intensive. In specialized fields such as medical imaging and remote sensing imagery, this requires specialized labelers, resulting in high labor and time costs. To reduce labeling costs, existing technologies utilize a small amount of manual annotation through efficient learning methods, such as semi-supervised learning, weakly supervised learning, and few-shot learning. Semi-supervised learning learns feature representations from a small number of labeled examples and uses this knowledge to guide learning on large amounts of unlabeled data. Weakly supervised learning utilizes incomplete or imprecise label information, managing label uncertainty through designed algorithms and extracting useful information from it. Few-shot learning focuses on learning with only a very small number of labeled examples, typically achieved through meta-learning and transfer learning techniques. While these methods reduce the reliance on large amounts of labeled data, they still require a certain amount of labeled data to train the model and struggle to achieve the performance of fully labeled learning.

[0004] In recent years, visual language models have combined image and text information to learn the association between image content and text descriptions on a large number of general datasets. This allows the models to understand the semantic connection between images and text, and achieve powerful zero-shot inference capabilities. This means that for image categories not seen during training, the models can classify target samples using textual cues. However, when the domain difference between the target task dataset and these general datasets is large, directly applying these models for inference still struggles to achieve high accuracy. Visual language models provide new insights for the automatic annotation of target task datasets, but the key to addressing the performance degradation caused by domain shift is how to adapt them to the target dataset in an unsupervised manner.

[0005] In view of this, an unsupervised image classification training method, system and medium based on a visual language model were developed to solve the above technical problems. Summary of the Invention

[0006] The purpose of the present invention is to disclose an unsupervised image classification training method, system and medium based on a visual language model.

[0007] The first object of the present invention is to provide an unsupervised image classification training method based on a visual language model.

[0008] The second object of the present invention is to provide an unsupervised image classification training system based on a visual language model.

[0009] A third object of the present invention is to provide a computer-readable storage medium.

[0010] To achieve the above first object of the invention, the present invention provides an unsupervised image classification training method based on a visual language model, comprising the following steps:

[0011] Step S1: For the image classification task, establish a training dataset D without manually annotated classification labels and a test dataset with classification labels annotated by human experts;

[0012] Step S2: Perform zero-shot inference on the training dataset based on the visual language model to obtain pseudo labels;

[0013] Step S3: Use the multi-view consistency strategy to filter the pseudo labels obtained in step S2 to obtain a preliminary reliable first training subset D ir ;

[0014] Step S4: For the first training subset D in step S3 ir , further screening is performed using the classification-clustering result consistency strategy to obtain the final reliable second training subset D l ;

[0015] Step S5: Using a high confidence cross-supervised learning method, the second training subset D containing pseudo labels is used l and the remaining unlabeled training subset D u Train an image classification model, where D u =DD l ;

[0016] Step S6: Save the image classification model trained in step S5 and test the samples in the test set to verify and evaluate the performance of the image classification model.

[0017] Preferably, the number of samples in the training dataset D is much larger than the number of samples in the test set.

[0018] Preferably, in step S2, the method for generating the pseudo-label comprises the following steps:

[0019] Step S21: N and C represent the number of samples and the number of categories in the training data set D respectively, X i Denotes the i-th training image, where i = 1, 2, ..., N; E img and E text Represent the image encoder and text encoder in the visual language model respectively; the image feature is represented by f i =E img (X i ) is obtained; the text prompt of the cth category is T c , text features are passed Obtain, among which To convert the text prompt into a high-dimensional feature segmenter, use p i ∈[0, 1] C Represents X i The classification probability vector, whose cth element p i，c Represents X i The probability of belonging to class c is obtained as follows:

[0020]

[0021] in, and τ represent the cosine similarity and temperature coefficient, respectively;

[0022] For each sample in the training dataset D, the predicted probability value is calculated using formula 1, and then the argmax operation is used to convert it into a pseudo label. The training dataset D is redefined as

[0023] Preferably, in step S3, the method for screening the pseudo-labels comprises the following steps:

[0024] Step S31:

[0025] For the same training image X i , use random image enhancement technology to generate K enhanced images, and use the method in step S2 to predict these K images respectively to obtain K prediction results, which are recorded as The average value is recorded as according to calculate The uncertainty is:

[0026]

[0027] in for The cth element in the training dataset D has an uncertainty value v for all samples. i Sort them in ascending order, and the Mth percentile is recorded as v M , taking it as a threshold of uncertainty value, using v M The samples in the training set are screened and the screened samples are used as the first training subset D ir , which is defined as:

[0028]

[0029] Preferably, in step S4, the second training subset D is obtained l The method comprises the following steps:

[0030] Step S41: D ir The features f corresponding to all images in i =E img (X i ) to cluster, the number of clusters is C, and O i Represents X i The cluster number to which it belongs is then mapped to the classification number using the Hungarian matching algorithm, and g(O i ) represents a bijective function for and O i The mapping between g(O i ) can be obtained by optimizing the following formula:

[0031]

[0032] Where δ is an impulse function, that is With g(O i ) are equal, the value is 1, otherwise the value is 0, by g(O i ) is solved, and then the and g(O i ) consistency, and screen out the second training subset D l , which is defined as:

[0033]

[0034] Preferably, in step S5, the method for verifying and evaluating the image classification model comprises the following steps:

[0035] Step S51: Use A and B to represent two different image classification networks, and use Represents the data augmentation set, training image X i use Two different random enhancements are performed, and the results are expressed as and Input the two into image classification network A and image classification network B respectively, and the results are expressed as and The calculation formula is as follows:

[0036]

[0037] For the second training subset D l In the sample, image classification network A and image classification network B learn different feature expressions through different supervision losses. Image classification network A uses the cross entropy loss function Image classification network B uses difficulty weighted loss function Both on a single training image X i The above definitions are:

[0038]

[0039] in is the training image X in the label i The probability of belonging to category c, and They are the corresponding probability values ​​predicted by image classification network A and image classification network B, α is a hyperparameter that controls the weight of difficult samples, and N l Indicates D l The number of samples in , the total supervision loss function is defined as:

[0040]

[0041] For the unlabeled training subset D u , using a high confidence mixed label weighted learning method, express and The result after dynamic mixing is defined as:

[0042]

[0043] Where β is a random mixing coefficient, which is randomly sampled from the uniform distribution U(0, 1) in each iteration, that is, β~U(0, 1). u High confidence hybrid pseudo-label loss on Defined as:

[0044]

[0045] Where γ is a probability threshold, yes The maximum value among the c probability values ​​of is the normalization factor, Lce is the cross entropy loss function,

[0046] Using the second training subset D l and the unlabeled training subset D u The total training loss function is:

[0047]

[0048] where λ is the equilibrium and The weight coefficients of the contributions of the two loss functions;

[0049] Based on the loss function Train an image classification model.

[0050] Preferably, in step S6, the test results are compared with the classification labels annotated by human experts to verify and evaluate the image classification model.

[0051] Based on the same inventive principle, in order to achieve the above-mentioned second invention object, the present invention provides an unsupervised image classification training system based on a visual language model, including a visual language model, a training dataset D without manually annotated classification labels, and a test set with classification labels annotated by human experts;

[0052] The training dataset D and the test set come from the same public dataset;

[0053] Perform zero-shot inference on the training dataset D based on the visual language model to obtain pseudo labels;

[0054] The pseudo labels are filtered by using the multi-view consistency strategy and the classification-clustering result consistency strategy to obtain the final reliable second training subset D l ;

[0055] A high-confidence cross-supervised learning method is used, using the second training subset D containing pseudo labels l and the remaining unlabeled training subset D u Train and obtain the image classification model, where D u =DD l ;

[0056] The image classification model is tested on samples in the test set to verify and evaluate the performance of the image classification model.

[0057] Preferably, the number of samples in the training dataset D is much larger than the number of samples in the test set.

[0058] Based on the same inventive principle, in order to achieve the above-mentioned third invention purpose, the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the unsupervised image classification training method based on the visual language model as described in the first invention.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] (1) Traditional deep learning methods usually require a large amount of labeled data to train the model. Even semi-supervised learning and small sample learning require a certain number of labeled samples. The present invention uses a pre-trained visual language model to generate pseudo labels for samples (images) in the training set. It can achieve high-accuracy classification without any manually labeled data, greatly reducing the cost of manual labeling.

[0061] (2) Due to the potential for domain shift between the training dataset of the visual language model and the downstream target dataset, the performance of existing visual language models directly used for zero-shot reasoning is reduced. The multi-view consistency strategy and classification-clustering consistency strategy proposed in this paper take into account sample uncertainty and similarity between samples, and can select reliable pseudo-labels, greatly reducing the negative impact of noisy labels on model training.

[0062] (3) Existing methods that use unlabeled images to help models perform domain adaptation rarely consider the quality of generated pseudo-labels, and incorrect pseudo-labels are detrimental to model training. This paper proposes high-confidence mixed pseudo-labels, which generate pseudo-labels for high-confidence unlabeled images and train the model using a random mixing method, which can enhance the model's generalization ability.

[0063] (4) A pre-trained visual language model (PLIP) is used to generate pseudo labels on the unlabeled training dataset D. Reliable pseudo labels are selected through a multi-view consistency strategy and a classification-clustering consistency strategy. Finally, a hybrid training is performed through a high-confidence cross-supervised learning method. Finally, a high-precision image classification model that is robust to the target domain is trained. The training process is unsupervised and adaptive. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a flowchart of the unsupervised image classification training method based on the visual language model of the present invention.

[0065] Figure 2 Flowchart of the training process of the present invention.

[0066] Figure 3 Schematic diagram of the classification-clustering consistency sample screening process of the present invention.

[0067] Figure 4This is a comparison of the classification results of the present invention with other methods on the LC25K dataset.

[0068] Figure 5 This is a schematic diagram of the computer media module of the present invention. DETAILED DESCRIPTION

[0069] The present invention is described in detail below with reference to the various embodiments shown in the accompanying drawings, but it should be noted that these embodiments are not limitations of the present invention, and any equivalent transformations or substitutions in functions, methods, or structures made by ordinary technicians in this field based on these embodiments are all within the scope of protection of the present invention.

[0070] The specific implementation process of the present invention is described below through multiple embodiments.

[0071] Example 1:

[0072] Ginseng Figures 1 to 4 As shown, this embodiment discloses a specific implementation of an unsupervised image classification training method based on a visual language model (hereinafter referred to as "method").

[0073] In this embodiment, the pathological image classification task is taken as an example. This embodiment is implemented in a computer with an Intel(R) Core(TM) i9-9900K 3.60GHz CPU, an NVidia GTX3090 GPU, and 24.0GB of memory. The programming language is Python, and the deep learning framework used is Pytorch (an open source deep learning computing library).

[0074] Ginseng Figure 1 and Figure 2 As shown in FIG, the unsupervised image classification training method based on the visual language model includes the following steps:

[0075] Step S1: For the image classification task, a training dataset D without manually annotated classification labels and a test set with classification labels annotated by artificial experts are established; specifically, the image samples in the training dataset D and the test set are both from the public dataset LC25K, which contains 25,000 pathological sections of the lungs and colon, with 5 categories: normal lung tissue, lung adenocarcinoma, lung squamous cell carcinoma, normal colon tissue, and colon adenocarcinoma tissue. The dataset is randomly divided into a training dataset D and a test set. The number of samples in the training dataset D is much larger than the number of samples in the test set. For example, the public dataset LC25K is randomly divided into 80% of the training dataset D and 20% of the test set, and the labels in the training dataset D are discarded to simulate a training dataset without manually annotated classification labels; the classification labels obtained by artificial expert annotation on the test set are used as the gold standard to verify and evaluate the performance of the image classification model.

[0076] Step S2: Zero-shot inference is performed on the training dataset based on the visual language model to obtain pseudo-labels. Specifically, a visual language model with a domain similar to the images in the training dataset D is selected. Zero-shot inference is performed using the PLIP visual language model, which has been pre-trained in the pathology image domain, to generate pseudo-labels. The weights of the PLIP pre-trained model are obtained from Hugging Face (an open-source artificial intelligence algorithm model library). The PLIP visual language model consists of an image encoder and a text encoder. Both the image encoder and the text encoder are Transformer structures. The Transformer structure includes an input embedding layer, an encoder, and a decoder component. The input embedding layer maps each element of the input sequence into a high-dimensional vector space to form an embedded representation. The encoder is composed of multiple identical encoding layers stacked together. Each layer includes a self-attention mechanism and a feedforward neural network. The self-attention mechanism enables information exchange between elements in the sequence. The decoder is similarly composed of multiple decoding layers, each layer incorporating self-attention, encoder-decoder attention, and a feedforward neural network. In addition, positional encoding is incorporated into the input embedding to preserve the position information of elements in the sequence.

[0077] In step S2, the method for generating the pseudo-label comprises the following steps:

[0078] Step S21: N and C represent the number of samples and the number of categories in the training data set D respectively, X i Denotes the i-th training image, where i = 1, 2, ..., N; E img and E text Respectively represent the image encoder and text encoder in the visual language model PLIP; the image feature is represented by f i =E img (X i ) is obtained; considering that there are C = 5 categories in the classification task of this embodiment, the text prompt of the cth category is T c = "an H&E stained {CLS} image", where {CLS} is the class name, and the text features are Obtain, among which To convert the text prompt into a high-dimensional feature segmenter, use p i ∈[0, 1] C Represents X i The classification probability vector, whose cth element p i，c Represents X i The probability of belonging to class c is obtained as follows:

[0079]

[0080] in, and τ represent the cosine similarity and temperature coefficient, respectively, preferably τ = 1.0;

[0081] For each sample in the training dataset D, the predicted probability value is calculated using formula 1, and then the argmax operation is used to convert it into a pseudo label. The training dataset D is redefined as Through step S2, the pre-trained visual language model is used to generate pseudo labels for the samples (images) in the training set, which can achieve high-accuracy classification without any manual annotation data, greatly reducing the cost of manual annotation.

[0082] Step S3: Use the multi-view consistency strategy to filter the pseudo labels obtained in step S2 to obtain a preliminary reliable first training subset D ir Specifically, the method for screening the pseudo label includes the following steps: Step S31: for the same training image X i , use random image enhancement technology (such as rotation, color transformation, etc.) to generate K enhanced images, such as K = 10, and use the method in formula 1 in step S2 to predict these K images respectively, and obtain K prediction results, which are recorded as The average value is recorded as according to calculate The uncertainty is:

[0083]

[0084] in for The cth element in the training dataset D has an uncertainty value v for all samples. i Sort them in ascending order, and the Mth percentile is recorded as v M , taking it as a threshold of uncertainty value, using v M The samples in the training set are screened and the screened samples are used as the first training subset D ir , which is defined as:

[0085]

[0086] Step S4: For the first training subset D in step S3 ir , further screening is performed using the classification-clustering result consistency strategy to obtain the final reliable second training subset D l Specifically, in step S4, the second training subset D l The method comprises the following steps:

[0087] Step S41: Use K-means++ algorithm to transform Dir The features f corresponding to all images in i =E img (X i ) to cluster, the number of clusters is C, and O i Represents X i The cluster number to which it belongs is then mapped to the classification number using the Hungarian matching algorithm, and g(O i ) represents a bijective function for and O i The mapping between g(O i ) can be obtained by optimizing the following formula:

[0088]

[0089] Where δ is an impulse function, that is With g(O i ) are equal, the value is 1, otherwise the value is 0, by g(O i ) is solved, and then the and g(O i ) consistency, and screen out the second training subset D l , which is defined as:

[0090]

[0091] Since there may be a domain shift between the training dataset of the visual language model and the downstream target dataset, the performance of the existing visual language model directly used for zero-shot reasoning is degraded. Figure 3 ,Through steps S3 and S4, the multi-view consistency strategy and ,classification-clustering consistency strategy proposed in this embodiment ,take into account the sample uncertainty and the similarity between samples, and ,can select reliable pseudo labels, greatly reducing the negative impact of ,noise labels on model training.

[0092] Step S5: Using a high confidence cross-supervised learning method, the second training subset D containing pseudo labels is used l and the remaining unlabeled training subset D u Train an image classification model, where D u =DD l Specifically, in step S5, the method for verifying and evaluating the image classification model includes the following steps:

[0093] Step S51: A and B represent two different image classification networks. In this embodiment, the ResNet50 structure is used. Represents a data augmentation set, such as random cropping, random flipping, random rotation, etc., training image X i use Two different random enhancements are performed, and the results are expressed as and Input the two into image classification network A and image classification network B respectively, and the results are expressed as and The calculation formula is as follows:

[0094]

[0095] For the second training subset D l In the sample, image classification network A and image classification network B learn different feature expressions through different supervision losses. Image classification network A uses the cross entropy loss function Image classification network B uses difficulty weighted loss function Both on a single training image X i The above definitions are:

[0096]

[0097] in is the training image X in the label i The probability of belonging to category c, and They are the corresponding probability values ​​predicted by image classification network A and image classification network B, α is a hyperparameter, such as α = 0.5, which controls the weight of difficult samples, and N l Indicates D l The number of samples in , the total supervision loss function is defined as:

[0098]

[0099] For the unlabeled training subset D u , using a high confidence mixed label weighted learning method, express and The result after dynamic mixing is defined as:

[0100]

[0101] Where β is a random mixing coefficient, which is randomly sampled from the uniform distribution U(0, 1) in each iteration, that is, β~U(0, 1). u High confidence hybrid pseudo-label loss on Defined as:

[0102]

[0103] Where γ = 0.9 is a probability threshold, yes The maximum value among the c probability values ​​of is the normalization factor, L ce is the cross entropy loss function,

[0104] Using the second training subset D l and the unlabeled training subset D u The total training loss function is:

[0105]

[0106] Where λ = 0.1 is the equilibrium and The weight coefficients of the contributions of the two loss functions;

[0107] Based on the loss function The image classification model is trained with a batch size of 128 images, of which 64 images come from the second training subset D l , and the other 64 images come from the unlabeled training subset D u , using stochastic gradient descent (SGD) optimizer, the learning rate of training is initialized to 10 -4 , and multiply by 0.1 after every 100 iterations, for a total of 300 iterations. This embodiment proposes high-confidence mixed pseudo-labels, generates pseudo-labels for high-confidence unlabeled images, and trains the model using a random mixing method to enhance the generalization ability of the model.

[0108] Step S6: Save the image classification model trained in step S5 and test it on samples in the test set to verify and evaluate the performance of the image classification model. Specifically, after training is completed, the image classification model parameters are used to predict the test set images, and the prediction results are compared with the true labels to evaluate the model performance.

[0109] Figure 4 This is a comparison of the classification results of this embodiment on the LC25K dataset with other methods. The bar chart represents PLIP (visual language model), CLIP (general visual-language model proposed by OpenAI in 2021), BioCLIP (general visual-language model in the biological field), BiomedCLIP (CLIP model specifically optimized for the biomedical field), PubMedCLIP (CLIP model fine-tuned in the medical field) and the present invention from left to right. The evaluation indicators include accuracy, F1-score and area under the curve (AUC). It can be seen that under the same circumstances without the need for manual labeling, the performance of the method of this embodiment is much higher than that of other methods.

[0110] Example 2:

[0111] This embodiment discloses a specific implementation of an unsupervised image classification training system based on a visual language model (hereinafter referred to as the "system").

[0112] An unsupervised image classification training system based on a visual language model, executing the unsupervised image classification training method based on a visual language model described in Example 1, comprising: a visual language model, a training dataset D without manually annotated classification labels, and a test dataset with manually annotated classification labels by experts;

[0113] The training dataset D and the test set come from the same public dataset;

[0114] Perform zero-shot inference on the training dataset D based on the visual language model to obtain pseudo labels;

[0115] The pseudo labels are filtered by using the multi-view consistency strategy and the classification-clustering result consistency strategy to obtain the final reliable second training subset D l ;

[0116] A high-confidence cross-supervised learning method is used, using the second training subset D containing pseudo labels l and the remaining unlabeled training subset D u Train and obtain the image classification model, where D u =DD l ;

[0117] The image classification model is tested on samples in the test set to verify and evaluate the performance of the image classification model.

[0118] Specifically, in this embodiment, the image samples in the training dataset D and the test set are both the public dataset LC25K, which contains 25,000 pathological sections of the lung and colon, with 5 categories: normal lung tissue, lung adenocarcinoma, lung squamous cell carcinoma, normal colon tissue, and colon adenocarcinoma tissue. The dataset is randomly divided into a training dataset D and a test set. The number of samples in the training dataset D is much larger than the number of samples in the test set. For example, the public dataset LC25K is randomly divided into 80% training dataset D and 20% test set, and the labels in the training dataset D are discarded to simulate the absence of manual labeling. The training dataset is annotated with classification labels; the classification labels obtained by manual expert annotation on the test set are used as the gold standard to verify and evaluate the performance of the image classification model; the pre-trained visual language model is used to generate pseudo labels for the samples (images) in the training set, which can achieve high-accuracy classification without any manually labeled data, greatly reducing the cost of manual annotation; the multi-view consistency strategy and classification-clustering consistency strategy proposed in this embodiment take into account the sample uncertainty and the similarity between samples, and can select reliable pseudo labels, greatly reducing the negative impact of noise labels on model training; based on the loss function The image classification model is trained with a batch size of 128 images, of which 64 images come from the second training subset D l , and the other 64 images come from the unlabeled training subset D u , using stochastic gradient descent (SGD) optimizer, the learning rate of training is initialized to 10 -4 , and multiply by 0.1 after every 100 iterations, for a total of 300 iterations. This embodiment proposes high-confidence mixed pseudo-labels, generates pseudo-labels for high-confidence unlabeled images, and trains the model using a random mixing method to enhance the generalization ability of the model.

[0119] The unsupervised image classification training system based on the visual language model disclosed in this embodiment has the same technical solutions as those in the first embodiment. Please refer to the description in the first embodiment and will not be repeated here.

[0120] Example 3:

[0121] Computer readable storage medium, see Figure 5 , which stores a computer program, which, when executed by a processor, implements the unsupervised image classification training method based on a visual language model as described in Example 1.

[0122] The computer-readable storage medium disclosed in this embodiment has the same technical solutions as those in the first embodiment. Please refer to the first embodiment and will not be repeated here.

[0123] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0124] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

[0125] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

[0126] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. An unsupervised image classification training method based on a visual language model, characterized in that: The following steps are involved: Step S1: For the image classification task, establish a training dataset D without manually annotated classification labels and a test dataset with classification labels annotated by human experts; Step S2: Perform zero-shot inference on the training dataset based on the visual language model to obtain pseudo labels; Step S3: Use the multi-view consistency strategy to filter the pseudo labels obtained in step S2 to obtain a preliminary reliable first training subset D ir ; Step S4: For the first training subset D in step S3 ir , further screening is performed using the classification-clustering result consistency strategy to obtain the final reliable second training subset D l ; Step S5: Using a high confidence cross-supervised learning method, the second training subset D containing pseudo labels is used l and the remaining unlabeled training subset D u Train an image classification model, where D u =DD l ; Step S6: Save the image classification model trained in step S5 and test the samples in the test set to verify and evaluate the performance of the image classification model.

2. The unsupervised image classification training method based on a visual language model according to claim 1, wherein: The number of samples in the training dataset D is much larger than the number of samples in the test set.

3. The unsupervised image classification training method based on a visual language model according to claim 1 or 2, characterized in that: In step S2, the method for generating the pseudo-label comprises the following steps: Step S21: N and C represent the number of samples and the number of categories in the training data set D respectively, X i Denotes the i-th training image, where i = 1, 2, ..., N; E img and E text Represent the image encoder and text encoder in the visual language model respectively; the image feature is represented by f i =E img (X i ) is obtained; the text prompt of the cth category is T c , text features are passed Obtain, among which To convert the text prompt into a high-dimensional feature segmenter, use p i ∈[0, 1] C Represents X i The classification probability vector, whose cth element p i，c Represents X i The probability of belonging to class c is obtained as follows: in, and τ represent the cosine similarity and temperature coefficient, respectively; For each sample in the training dataset D, the predicted probability value is calculated using formula 1, and then the argmax operation is used to convert it into a pseudo label. The training dataset D is redefined as 4. The unsupervised image classification training method based on a visual language model according to claim 3, wherein: In step S3, the method for screening the pseudo-labels includes the following steps: Step S31: For the same training image X i , use random image enhancement technology to generate K enhanced images, and use the method in step S2 to predict these K images respectively to obtain K prediction results, which are recorded as The average value is recorded as according to calculate The uncertainty is: in for The cth element in the training dataset D has an uncertainty value v for all samples. i Sort them in ascending order, and the Mth percentile is recorded as v M , taking it as a threshold of uncertainty value, using v M The samples in the training set are screened and the screened samples are used as the first training subset D ir , which is defined as:

5. The unsupervised image classification training method based on a visual language model according to claim 4, characterized in that: In step S4, the second training subset D is obtained l The method comprises the following steps: Step S41: D ir The features f corresponding to all images in i =E img (X i ) to cluster, the number of clusters is C, and O i Represents X i The cluster number to which it belongs is then mapped to the classification number using the Hungarian matching algorithm, and g(O i ) represents a bijective function for and O i The mapping between g(O i ) can be obtained by optimizing the following formula: Where δ is an impulse function, that is With g(O i ) are equal, the value is 1, otherwise the value is 0, by g(O i ) is solved, and then the and g(O i ) consistency, and select the second training subset D l , which is defined as:

6. The unsupervised image classification training method based on a visual language model according to claim 5, characterized in that: In step S5, the method for verifying and evaluating the image classification model includes the following steps: Step S51: Use A and B to represent two different image classification networks, and use Represents the data augmentation set, training image X i use Two different random enhancements are performed, and the results are expressed as and Input the two into image classification network A and image classification network B respectively, and the results are expressed as and The calculation formula is as follows: For the second training subset D l In the sample, image classification network A and image classification network B learn different feature expressions through different supervision losses. Image classification network A uses the cross entropy loss function Image classification network B uses difficulty weighted loss function Both on a single training image X i The above definitions are: in is the training image X in the label i The probability of belonging to category c, and They are the corresponding probability values ​​predicted by image classification network A and image classification network B, α is a hyperparameter that controls the weight of difficult samples, and N l Indicates D l The number of samples in , the total supervision loss function is defined as: For the unlabeled training subset D u , using a high confidence mixed label weighted learning method, express and The result after dynamic mixing is defined as: Where β is a random mixing coefficient, randomly sampled from the uniform distribution U(0, 1) in each iteration, that is, β~U(0, 1), in the unlabeled training subset D u High confidence hybrid pseudo-label loss on Defined as: Where γ is a probability threshold, yes The maximum value among the c probability values ​​of is the normalization factor, L ce is the cross entropy loss function, Using the second training subset D l and the unlabeled training subset D u The total training loss function is: where λ is the equilibrium and The weight coefficients of the contributions of the two loss functions; Based on the loss function Train an image classification model.

7. The unsupervised image classification training method based on a visual language model according to claim 6, characterized in that: In step S6, the test results are compared with the classification labels annotated by human experts to verify and evaluate the image classification model.

8. Unsupervised image classification training system based on visual language model, characterized by: Includes a visual language model, a training dataset D without manually annotated classification labels, and a test set with classification labels annotated by human experts; The training dataset D and the test set come from the same public dataset; Perform zero-shot inference on the training dataset D based on the visual language model to obtain pseudo labels; The pseudo labels are filtered by using the multi-view consistency strategy and the classification-clustering result consistency strategy to obtain the final reliable second training subset D l ; A high-confidence cross-supervised learning method is used, using the second training subset D containing pseudo labels l and the remaining unlabeled training subset D u Train and obtain the image classification model, where D u =DD l ; The image classification model is tested on samples in the test set to verify and evaluate the performance of the image classification model.

9. The unsupervised image classification training system based on a visual language model according to claim 8, characterized in that: The number of samples in the training dataset D is much larger than the number of samples in the test set.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the unsupervised image classification training method based on a visual language model as described in any one of claims 1 to 7 is implemented.

Images