Image classification system and method based on large-scale pre-trained model

Abstract

The application discloses an image classification system and method based on a large-scale pre-training model, wherein an image is input into a first branch image pre-training model and a second branch auxiliary pre-training model for feature extraction; a predictor is connected to an output part of the auxiliary pre-training model, and a few-sample training is performed on the predictor based on a multi-branch training strategy and an initialization strategy of sample features; pre-training image features of the first branch and initial image classification results are fused with few-sample trained auxiliary image classification results logit of the auxiliary pre-training model and the predictor of the second branch based on sample uncertainty kappa; and a final image classification result is obtained by processing the fusion result by using a Softmax layer. Compared with the prior art, the application realizes an efficient and accurate image recognition task solution by using a pre-training large model.

Description

Technical Field

[0001] This invention relates to neural networks and image recognition technology, and in particular to an image classification system and method based on a large-scale pre-trained model. Background Technology

[0002] Traditional image recognition methods are often limited by specific scenarios and the need for large amounts of labeled data, resulting in poor generalization performance for downstream tasks lacking sufficient labeled samples. However, the emergence of large-scale pre-trained models in recent years has provided new ideas and possibilities for solving this problem. These models have achieved significant results in the field of natural language processing. They are pre-trained on massive amounts of text data, learning rich semantic representations. In recent years, this pre-training strategy has been extended to the image domain, with large-scale pre-trained models, such as the CLIP model, bringing new breakthroughs to image recognition tasks. By pre-training on large-scale image data, these models can learn general image features and demonstrate strong potential in various downstream tasks.

[0003] In practical applications, data scarcity is a frequent challenge, especially in specific domains or tasks. To overcome this challenge of limited sample data, few-shot fine-tuning techniques have emerged. This technique can be applied to large-scale pre-trained models, fine-tuning them with a small number of labeled samples to better adapt them to the target task. This strategy effectively utilizes the general features learned by pre-trained models on large-scale data, providing a feasible solution for image recognition tasks in few-shot scenarios. Few-shot fine-tuning techniques in image recognition are of great significance for improving the application performance of image recognition in real-world scenarios.

[0004] To achieve few-sample fine-tuning on large-scale pre-trained models, the current technical approaches can be summarized into the following two categories:

[0005] The first approach involves incorporating learnable sequences at different locations within a large model. These learnable sequences are added to the input images or text labels, and then the images or text combined with the learnable sequences are fed into the model for training. During training, the entire pre-trained model is kept frozen, and adjustments are made only to these specific sequences to allow the model to acquire information from the target data. However, this approach typically requires backpropagation of the entire model, resulting in high computational complexity and long processing time. Furthermore, existing approaches of this type also exhibit poor performance on the target task.

[0006] The second approach involves adding different types of lightweight trainable adapters to the model's output. This approach offers higher training efficiency, but is prone to overfitting due to the limited sample size. Furthermore, existing methods, after adding adapters, do not fully utilize the original information from the pre-trained large model, thus limiting model performance at the current stage. Summary of the Invention

[0007] To address the shortcomings of the existing technologies, this invention aims to propose an image classification system and method based on a large-scale pre-trained model. During the fine-tuning process, auxiliary features, multi-branch training strategies, and uncertainty fusion strategies between branches are introduced to achieve image classification on downstream datasets based on the large-scale pre-trained model.

[0008] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0009] An image classification system based on a large-scale pre-trained model includes:

[0010] The first branch includes a pre-trained model branch, and the second branch includes an auxiliary pre-trained model and a predictor connected to the output of the auxiliary pre-trained model, an uncertainty fusion module, and a Softmax layer.

[0011] The image pre-training model is used to extract features from the input image to obtain an initial image classification result; the auxiliary pre-training model is used to extract auxiliary features from the input image to enhance the training process, obtaining an auxiliary image classification result logit; the predictor is set at the output of the auxiliary pre-training model and performs few-shot training of the auxiliary image classification result logit based on a multi-branch training strategy and a sample feature initialization strategy; the uncertainty fusion module 500 is used to fuse the initial image classification result obtained by the first branch image pre-training model with the few-shot trained auxiliary image classification result logit obtained by the second branch auxiliary pre-training model and the predictor, and the fused image classification result is transmitted to the Softmax layer; the expression of the fused image classification result is as follows:

[0012]

[0013] Where β represents the hyperparameter controlling the proportion used in the fusion of the two branches, f Aux This represents the image features extracted by the auxiliary pre-trained model. This represents the predictor after initialization.

[0014] The Softmax layer is used to obtain the final predicted value for image classification based on the fused calculation results.

[0015] An image classification method based on a large-scale pre-trained model includes:

[0016] Step 1: Simultaneously input the image into the image pre-training model of the first branch and the auxiliary pre-training model of the second branch for feature extraction. Use the image pre-training model to extract features from the input image to obtain the initial image classification result. Use the auxiliary pre-training model to extract auxiliary features from the input image to enhance the training process and obtain the auxiliary image classification result logit.

[0017] Step 2: Connect the predictor to the output of the auxiliary pre-trained model, and perform few-sample training on the predictor based on the multi-branch training strategy and the sample feature initialization strategy. The process is as follows:

[0018] Step 2-1: In scenarios with few samples, initialize the predictor using an initialization strategy based on sample features;

[0019] Step 2-2: Train the predictor using a multi-branch training strategy, that is, use the cross-entropy loss function l of the image classification results of the first and second branches. Fusion The loss function l in the second branch auxiliary pre-trained model Aux The predictor is trained to obtain the total loss function l. total :

[0020] Step 3: The initial image classification result obtained from the image pre-training model of the first branch based on the sample uncertainty κ is fused with the logit of the auxiliary image classification result trained with few samples obtained from the auxiliary pre-training model and predictor of the second branch. The fused image classification result is then transmitted to the Softmax layer. The expression of the fused image classification result is as follows:

[0021]

[0022] Where β represents the hyperparameter controlling the proportion used in the fusion of the two branches, f Aux This represents the image features extracted by the auxiliary pre-trained model. This represents the predictor after initialization.

[0023] Step 4: Transfer the fused image classification results to the Softmax layer to obtain the final image classification results.

[0024] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0025] 1) It improves upon the impact of traditional methods, which heavily rely on hyperparameters that require manual adjustment, on model performance;

[0026] 2) Relying on the prediction training results of the final fusion of the two pre-trained model branches, image data classification is achieved on the downstream dataset;

[0027] 3) By utilizing pre-trained large models and a small number of samples for specific domains, such as identifying rare samples that are less frequently encountered during training in autonomous driving scenarios, and analyzing and statistically analyzing rare plants and animals in the field of ecological protection, it provides efficient and accurate image recognition task solutions. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the image classification system framework based on a large-scale pre-trained model according to the present invention;

[0029] Figure 2 This is a schematic diagram of the image classification method based on a large-scale pre-trained model according to the present invention. Detailed Implementation

[0030] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] like Figure 1 As shown, the image classification system based on a large-scale pre-trained model of the present invention mainly consists of a first branch including a pre-trained model branch 200, a second branch including an auxiliary pre-trained model 300, a predictor 400 connected to the output of the auxiliary pre-trained model 500, an uncertainty-based fusion module 500, and a softmax layer 600. Wherein:

[0032] The image pre-training model 200 of the first branch and the auxiliary pre-training model 300 of the second branch simultaneously extract features from the input image 100. The image pre-training model 200 is used to extract features from the input image 100, and the obtained image features are used as the initial image classification result. The auxiliary pre-training model 300 is used to extract auxiliary features from the input image 100 (to enhance the training process), and obtains auxiliary image classification results logits. The predictor is set at the output of the auxiliary pre-training model and performs few-shot training based on the multi-branch training strategy and the sample feature initialization strategy. The uncertainty fusion module 500 is used to fuse the initial image classification result obtained by the image pre-training model of the first branch with the auxiliary image classification result logits obtained by the auxiliary pre-training model and the predictor of the second branch after few-shot training. The fused image classification result is transmitted to the Softmax layer 600.

[0033] The expression for the fused image classification result is as follows:

[0034]

[0035] Where β represents the hyperparameter controlling the proportion used in the fusion of the two branches, f Aux This represents the image features extracted by the auxiliary pre-trained model. This represents the predictor after initialization.

[0036] The Softmax layer 600 is used to obtain the final predicted value for image classification based on the fused calculation results.

[0037] Specifically, the input image features are RGB image features.

[0038] Specifically, the image pre-training model adopts the Contrastive Language-Image Pre-Training (CLIP) model.

[0039] Specifically, the auxiliary pre-training model can be a model such as MAE, DINO, MoCov3, SparK, or MILAN, which can achieve accuracies of 65.49, 68.32, 69.35, 63.56, and 69.24 on the ImageNet-1K dataset, respectively. Logits refer to the output of the layer preceding the softmax layer in the neural network.

[0040] Specifically, the auxiliary pre-trained model is a pre-trained model that has never been seen before and contains labels of the real target dataset, in order to ensure the reliability of the evaluation results.

[0041] Specifically, the predictor employs a linear classifier based on a sample feature initialization strategy and a multi-branch training strategy. The predictor itself can be designed in many ways, such as an MLP structure, a cache structure, or an LP (linear structure); experimental analysis shows that their performance differences are not significant, and this invention uses the linear structure with the lowest computational cost. However, the linear structure suffers from weak training stability; therefore, a sample initialization strategy is introduced to alleviate this problem.

[0042] In scenarios with few samples, the predictor is initialized using an initialization strategy based on sample features. The initialized predictor... The expression is as follows:

[0043]

[0044]

[0045] in, Represents the initialized linear predictor Let m1, m2, ..., m be the j-th feature in the i-th category. C Image features are calculated for the auxiliary pre-trained models corresponding to categories 1, 2, ..., C, where C represents the total number of categories and N represents the total number of samples.

[0046] The sample feature-based initialization strategy obtains image features from the input image features used for training through feature extraction of the auxiliary pre-trained model 200, and uses the mean of these image features to initialize the predictor, thereby improving the stability and model performance during training. A linear layer is used to reduce computational cost.

[0047] Structurally, the predictor of this invention employs a linear layer that helps reduce computational costs and utilizes an initialization strategy based on sample average features to improve the stability of the predictor during small sample training.

[0048] Furthermore, the predictor is trained using a multi-branch training strategy, specifically by using the cross-entropy loss function l from the image classification results of the first and second branches. Fusion The loss function l in the second branch auxiliary pre-trained model Aux The predictor is trained to obtain the total loss function l. total .

[0049] The loss function l in the second branch of the auxiliary pre-trained model Aux The expression is as follows:

[0050]

[0051] Where g(·) represents the Softmax function, s j The expression for the total loss function of the logit predictor for the j-th training sample is as follows:

[0052] l total =(1-λ)l Aux +λl Fusion

[0053] Where λ represents the hyperparameter used to balance l Fusion and l Aux , l Fusion The cross-entropy loss function represents the image classification results of the first and second branches.

[0054] Compared to the traditional training method using a single loss function, in order to better train the classifier in the auxiliary branch, we designed a training strategy that fuses multiple loss function values.

[0055] In the uncertainty fusion module 500, the initial image classification result obtained from the image pre-training model of the first branch is fused with the logit of the auxiliary image classification result trained with few samples obtained from the auxiliary pre-training model and predictor of the second branch. The kurtosis (fourth moment) of the sample features is used to calculate the image classification prediction uncertainty. The expression for the uncertainty κ of the image classification prediction is as follows:

[0056]

[0057] Where s0 represents the training result of the original image pre-trained model (CLIP), μ and σ represent the mean and variance of s0, respectively, and ρ represents the parameter controlling the uncertainty.

[0058] The fusion result expression based on the two branches of sample uncertainty κ is as follows:

[0059]

[0060] Where β represents the hyperparameter controlling the proportion used in the fusion of the two branches, f Aux This represents the image features extracted by the auxiliary pre-trained model. This represents the predictor after initialization.

[0061] Uncertainty κ is used to balance the impact of hyperparameters on the image pre-trained model (CLIP).

[0062] Example 2:

[0063] The image classification method based on a large-scale pre-trained model of the present invention includes the following steps:

[0064] Step 1: Simultaneously input the image into the first branch image pre-training model and the second branch auxiliary pre-training model for feature extraction; use the image pre-training model 200 to extract features from the input image 100, and use the obtained image features as the initial image classification result; use the auxiliary pre-training model 300 to extract auxiliary features from the input image 100, and obtain the auxiliary image classification result logit.

[0065] Step 2: Connect the predictor to the output of the auxiliary pre-trained model, and perform few-sample training on the predictor based on the multi-branch training strategy and the sample feature initialization strategy. The process is as follows:

[0066] Step 2-1: In scenarios with few samples, the predictor is initialized using an initialization strategy based on sample features. The initialized predictor... The expression is as follows:

[0067]

[0068]

[0069] in, Represents the initialized linear predictor Let m1, m2, ..., m be the j-th feature in the i-th category. C Image features are calculated for the auxiliary pre-trained models corresponding to categories 1, 2, ..., C, where C represents the total number of categories and N represents the total number of samples.

[0070] Step 2-2: Train the predictor using a multi-branch training strategy, that is, use the cross-entropy loss function l of the image classification results of the first and second branches. Fusion The loss function l in the second branch auxiliary pre-trained model Aux The predictor is trained to obtain the total loss function l. total :

[0071] The loss function l in the second branch of the auxiliary pre-trained model Aux The expression is as follows:

[0072]

[0073] Where g(·) represents the Softmax function, s j Let logit be the value of the j-th training sample. The total loss function of the predictor is expressed as follows:

[0074] l total =(1-λ)l Aux +λl Fusion

[0075] Where λ represents the hyperparameter used to balance l Fusion and l Aux , l Fusion The cross-entropy loss function represents the image classification results of the first and second branches.

[0076] Step 3: The initial image classification result obtained by the image pre-training model of the first branch based on the sample uncertainty κ is fused with the auxiliary image classification result obtained by the auxiliary pre-training model and the predictor of the second branch through few-sample training (logit). The fused image classification result is then transmitted to the Softmax layer.

[0077] The uncertainty of image classification prediction is calculated using the kurtosis (fourth moment) of sample features. The expression for the uncertainty κ of image classification prediction is as follows:

[0078]

[0079] Where s0 represents the training result of the original image pre-trained model (CLIP), μ and σ represent the mean and variance of s0, respectively, and ρ represents the parameter controlling the uncertainty.

[0080] The expression for the fused image classification result is as follows:

[0081]

[0082] Where β represents the hyperparameter controlling the proportion used in the fusion of the two branches, f Aux This represents the image features extracted by the auxiliary pre-trained model. This represents the predictor after initialization.

[0083] Step 4: Transfer the fusion result to the Softmax layer to obtain the final image classification result.

[0084] Therefore, this invention innovates in three aspects of the small-sample fine-tuning method based on large-scale pre-trained models. Here, we provide a qualitative analysis of the three main innovations:

[0085] First, different auxiliary pre-trained models can provide more diverse information. Utilizing appropriate auxiliary pre-trained models to provide supplementary information is beneficial for assisting large pre-trained models in accomplishing a wide variety of target tasks.

[0086] Secondly, the predictor based on a multi-branch training strategy and sample feature initialization improves the stability of model training by initializing the predictor using the mean of the features of the training samples, and has a more significant advantage when the number of samples is small. Furthermore, the multi-branch training strategy provides information about auxiliary branches, allowing these auxiliary branches to be trained more thoroughly. This, in turn, can significantly improve the model's classification performance.

[0087] Finally, based on the uncertainty-based fusion strategy, the confidence of the pre-trained large model in the classification results is calculated using the kurtosis of the samples, which improves the robustness of the model and alleviates the obstacle of hyperparameters being too sensitive and affecting the model performance during the multi-branch fusion process.

[0088] The AMU fine-tuning method proposed in this invention achieves higher classification accuracy with less computation in few-shot image classification tasks based on large model fine-tuning. This result can be validated on public datasets such as ImageNet-1k. In this example, the ImageNet-1K, StanfordCars, Caltech101, UCF101, Flowers102, Food101, DTD, EuroSAT, FGVCAircraft, OxfordPets, and SUN397 datasets were used for thorough validation. The network model was trained on the ImageNet-1K dataset using the AdamW optimization algorithm and a cosine annealing training strategy. The weight decay parameter was set to 1e-4, the training batch size was 8, the initial learning rate was 0.001, and the model training iterations were 50.

[0089] It should be noted that the above content is only used to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Simple modifications or equivalent substitutions made by those skilled in the art to the technical solution of the present invention do not depart from the essence and scope of protection of the technical solution of the present invention.

Claims

1. An image classification system based on a large-scale pre-trained model, characterized in that, include: The first branch includes a pre-trained model branch, and the second branch includes an auxiliary pre-trained model and a predictor connected to the output of the auxiliary pre-trained model, an uncertainty fusion module, and a Softmax layer. The image pre-training model is used to extract features from the input image to obtain an initial image classification result; the auxiliary pre-training model is used to extract auxiliary features from the input image to obtain an auxiliary image classification result logit; the predictor is set at the output of the auxiliary pre-training model and performs few-shot training of the auxiliary image classification result logit based on a multi-branch training strategy and a sample feature initialization strategy; the uncertainty fusion module is used to fuse the initial image classification result obtained by the first branch image pre-training model with the few-shot trained auxiliary image classification result logit obtained by the second branch auxiliary pre-training model and the predictor, and the fused image classification result is transmitted to the Softmax layer; The expression for the fused image classification result is as follows: in, This represents a hyperparameter that controls the proportion used in merging the two branches. This represents the image features extracted by the auxiliary pre-trained model. This represents the predictor after initialization. Indicates sample uncertainty. This represents the training result of the pre-trained model based on the original image; The Softmax layer is used to obtain the final predicted value for image classification based on the fused calculation results.

2. The image classification method based on a large-scale pre-trained model according to claim 1, characterized in that, The initialized predictor The expression is as follows: in, Represents the initialized linear predictor , Indicates the first The first in the category One characteristic, Indicates 1, 2, ... Image features are calculated by the auxiliary pre-trained model corresponding to each category. Indicates the total number of categories in the sample. Indicates the total number of samples. Indicates the first The logit of the training sample, after feature initialization by a linear predictor, for the ... logit of each training sample The expression for the calculation process is as follows: 。 3. The image classification method based on a large-scale pre-trained model according to claim 1, characterized in that, The loss function in the second branch of the auxiliary pre-trained model The expression is as follows: in, This represents a Softmax function. Indicates the total number of categories in the sample. Indicates the total number of samples. Indicates the first The logit of each training sample, and the expression for the total loss function of the predictor are as follows: in, Indicates hyperparameters, The cross-entropy loss function represents the image classification results of the first and second branches.

4. An image classification method based on a large-scale pre-trained model, characterized in that, include: Step 1: Simultaneously input the image into the image pre-training model of the first branch and the auxiliary pre-training model of the second branch for feature extraction. Use the image pre-training model to extract features from the input image to obtain the initial image classification result. Use the auxiliary pre-training model to extract auxiliary features from the input image to enhance the training process and obtain the auxiliary image classification result logit. Step 2: Connect the predictor to the output of the auxiliary pre-trained model, and perform few-sample training on the predictor based on the multi-branch training strategy and the sample feature initialization strategy. The process is as follows: Step 2-1: In scenarios with few samples, initialize the predictor using an initialization strategy based on sample features; Step 2-2: Train the predictor using a multi-branch training strategy, that is, use the cross-entropy loss function of the image classification results of the first and second branches. Loss function in the second branch of the auxiliary pre-trained model The predictor is trained to obtain the total loss function. : Step 3: Based on sample uncertainty The initial image classification result obtained from the image pre-training model of the first branch is fused with the logit-based auxiliary image classification result obtained from the auxiliary pre-training model and predictor of the second branch, which has been trained with few samples. The fused image classification result is then passed to the Softmax layer. The expression for the fused image classification result is as follows: in, This represents a hyperparameter that controls the proportion used in merging the two branches. This represents the image features extracted by the auxiliary pre-trained model. This represents the predictor after initialization. Indicates sample uncertainty. This represents the training result of the pre-trained model based on the original image; Step 4: Transfer the fused image classification results to the Softmax layer to obtain the final image classification results.

5. The image classification method based on a large-scale pre-trained model according to claim 1, characterized in that, The initialized predictor The expression is as follows: in, Represents the initialized linear predictor , Indicates the first The first in the category One characteristic, For 1, 2, ... Image features are calculated by the auxiliary pre-trained model corresponding to each category. Indicates the total number of categories in the sample. Indicates the total number of samples. Indicates the first The logit of the training sample, after feature initialization by a linear predictor, for the ... logit of each training sample The expression for the calculation process is as follows: 。 6. The image classification method based on a large-scale pre-trained model according to claim 1, characterized in that, The loss function in the second branch of the auxiliary pre-trained model The expression is as follows: in, This represents the Softmax function. Indicates the first Logit of each training sample, Indicates the total number of categories in the sample. Indicates the total number of samples. Indicates the first The logit of each training sample, and the expression for the total loss function of the predictor are as follows: in, Indicates hyperparameters, The cross-entropy loss function represents the image classification results of the first and second branches.

Images