An incremental small sample image classification method based on bias correction

Abstract

The present invention discloses an incremental small-sample image classification method based on bias correction. The method utilizes a transformation matrix to transform a biased prediction distribution into an unbiased prediction distribution, and fine-tunes the model's final layer to adapt to the incremental data representation to assist in bias correction. The transformation matrix has a random diagonal dominant column and is a learnable transformation matrix, capable of transforming a prediction distribution biased toward the base class from an incremental class into a normal, unbiased prediction distribution. Compared to existing related technologies, the present invention has the advantages of being free from feature space constraints and catastrophic forgetting caused by continuous model parameter updates during the incremental phase. By giving greater attention to incremental task data, higher classification accuracy results are achieved for both incremental task data and all task data.

Description

Technical Field

[0001] The present invention relates to the technical fields of small sample image classification and incremental learning technology of computer vision, and in particular to the construction of an incremental small sample image classification method based on deviation correction. Background Art

[0002] The continuous development of deep learning has achieved remarkable results in the field of computer vision, such as image classification, object detection, and semantic segmentation, greatly improving industrial production efficiency and providing great convenience for people's daily lives. However, current applications rely on large amounts of labeled data to train models. In practice, these applications often face the following bottlenecks: First, collecting large amounts of data requires a significant expenditure of human and economic resources, and some areas, such as those involving privacy and security, simply cannot collect large amounts of data. Second, manually labeling large amounts of data is not only time-consuming and slows down application updates and iterations, but also prone to inaccurate labeling, which negatively impacts results. Third, traditional methods require retraining the model with each iteration, which consumes a significant amount of time. Therefore, how to utilize only a small amount of labeled data without retraining the entire model during iteration has become an urgent problem. Incremental small-sample image classification technology is a solution to this dilemma.

[0003] Incremental small sample sizes require that a model not only correctly classify a continuous stream of small amounts of data but also maintain its ability to classify older data. Traditional classification methods, faced with continuous small streams of data, must retrain each new batch of data by combining it with previously trained data, resulting in significant training overhead. Incremental small sample sizes, however, enable the model to correctly classify new data using only these data sets while maintaining its ability to classify the base class data.

[0004] Current incremental small-sample image classification methods mainly focus on the research of feature space reservation and regularization methods. The former gradually reduces the model's ability to represent incremental data as the number of increments increases, while the latter has strong forgetting based on regularization. In addition to the above defects, existing methods focus on preventing catastrophic forgetting and ignore the importance of learning new knowledge. Summary of the Invention

[0005] The purpose of the present invention is to provide an incremental small sample image classification method based on bias correction to address the shortcomings of the existing technology. It adopts a modeling method that introduces a transformation matrix. First, in the incremental learning stage, the last layer of the model is fine-tuned with the distillation loss and a very small cache is used to prevent forgetting. Secondly, the transformation matrix is used to convert the predicted distribution of the base class and incremental confusion into the correct predicted distribution.

[0006] The object of the present invention is achieved like this:

[0007] This paper proposes an incremental small-shot image classification method based on bias correction. It features a modeling approach that introduces a transformation matrix. First, during the incremental learning phase, the final model layer is fine-tuned using a distillation loss and a cache is used to prevent forgetting. This cache stores one image for each base class. Second, the transformation matrix is used to transform the predicted distribution of the base class and incremental class confusion into the correct predicted distribution. The specific steps are as follows:

[0008] Step 1: Dataset preparation and preprocessing

[0009] Collect image data to be classified and divide it into different tasks according to categories. The first task requires a large amount of data and uses all the data for each category. Subsequent incremental tasks are set according to the N-way K-shot method, where N is the number of categories and K is the number of samples in each category.

[0010] Step 2: Base task model construction and training

[0011] The base class task model consists of a base class feature extraction network and a base class classifier. The base class feature extraction network uses Resnet-18, the base class classifier uses a fully connected layer, and the base class task model uses a cross-entropy loss optimization model. This step aims to use a large amount of existing labeled data to train a base class feature extraction network.

[0012] Step 3: Incremental task model construction

[0013] 3-1: Constructing an incremental feature extraction network: fix the first three layers of the base feature extraction network, and use the last layer to adapt to incremental tasks;

[0014] 3-2: Initialize the transformation matrix M with a dimension equal to the sum of the number of incremental classes and the number of base classes. Use the sigmoid nonlinear activation function to activate it. Replace the diagonal elements of the transformation matrix corresponding to the base class with the diagonal elements corresponding to the incremental class. The diagonal elements of the base class must be smaller than the diagonal elements of the incremental class.

[0015] Step 4: Incremental task training

[0016] 4-1: Extract the image features of base class data using the base class feature extraction network and base class training data base , and calculate the prototype point of each category of the base class, the formula is as follows:

[0017]

[0018] in is the prototype point of the cth class of the base class, and K is the number of pictures of each category of the base class;

[0019] 4-2: Extract incremental image features feat using the incremental feature extraction network built in 3-1 and incremental training data novel ;

[0020] 4-3: Use the incremental image features extracted in 4-2 to calculate the incremental image prototype point. The formula is as follows:

[0021]

[0022] in is the prototype point of the cth class of the incremental class, N is the number of pictures of each category of the incremental class; Merge with the base class prototype point obtained in 4-1, It is the prototype point of all categories and is used for classification;

[0023] 4-4: Using cosine similarity as a metric, we use the prototype points obtained in 4-3 and the features obtained in 4-2 to classify and obtain a predicted distribution where the incremental class is biased towards the base class. Indicates the probability that the predicted sample X is predicted to be category Y;

[0024] 4-5: Use the conversion matrix of 3-2 to correct the predicted distribution of 4-4,

[0025]

[0026] The corrected distribution P(T|X) is an unbiased predictive distribution.

[0027] Compared with existing methods, the incremental small sample image classification method based on deviation correction proposed in the present invention can be applied in real time in incremental tasks because it uses a transformation matrix M for real-time correction, and the average accuracy of the incremental class is improved by 10 percentage points compared with before correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Base task training flow chart;

[0029] Figure 2 Incremental task training flow chart;

[0030] Figure 3 Transformation matrix update flow chart;

[0031] Figure 4 Flowchart of the present invention;

[0032] Figure 5 Experimental results diagram;

[0033] Figure 6 Visualization of the bias correction effect. DETAILED DESCRIPTION

[0034] In order to facilitate understanding of the present invention, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0035] See Figure 1 The present invention first uses a large amount of existing annotated data to train a base class model, aiming to use the existing large amount of annotated data to train a base class feature extractor. This feature extractor has the ability to represent the features of the image, laying a solid foundation for the subsequent incremental learning of a small amount of N-way K-shot data;

[0036] B0: Use data augmentation to perform a series of transformations on the input image, aiming to make the feature representation generalization performance of the base class feature extractor more powerful. Specifically, Randaugment is used as a data augmentation method to perform a large number of transformations on the original input image, such as flipping, center cropping, mosaicking, etc.

[0037] B1: Using Resnet-18 as the feature extraction module can effectively extract image features;

[0038] B2: Use the fully connected layer as the classification module to classify the image.

[0039] See Figure 2 , when training incremental tasks, use Figure 1 The trained base class feature extractor is used to initialize the incremental class feature extractor, and one image of each base class is stored as backtracking data to prevent catastrophic forgetting. Prototype point classification is used instead of fully connected layer classification to avoid greater deviations caused by classifier updates. Finally, the transformation matrix is used to correct the predicted distribution.

[0040] I0: Perform data enhancement on incremental task data and a small amount of backtracking data. The specific implementation is the same as B0.

[0041] I1: During feature extraction, we fix the first three layers of the incremental feature extractor and only fine-tune the last layer. We also use distillation loss to prevent catastrophic forgetting. The distillation loss is as follows:

[0042] l kd =|feat base -feat incre |2 (1);

[0043] Among them, base and feat incre These are the features extracted from the image by the base class feature extractor and the incremental class feature extractor respectively;

[0044] I2: Extract the image features of the base class data using the base class feature extraction network and base class training data base, and calculate the prototype point of each category of the base class, the formula is as follows:

[0045]

[0046] in is the prototype point of the cth class of the base class, K is the number of images of each category of the base class, the incremental class feature extraction network and the incremental class training data extract the incremental class image features feat novel , and calculate the prototype point of each category of the incremental class, as shown below:

[0047]

[0048] in is the prototype point of the cth category of the incremental class, K is the number of pictures of each category of the incremental class, and the prototype points of the base class and the incremental class are merged.

[0049] I3: Combine the image features extracted for the current task with the prototype points obtained in I2 Calculate cosine similarity to get predicted distribution The prediction distribution at this time will be heavily biased towards the base class. This is because the base class feature extractor is mostly trained with base class data. If the incremental class data is used to fine-tune the incremental class feature extractor initialized by the base class feature extractor, it will cause severe overfitting due to the lack of data.

[0050] I4: Use the transformation matrix M to transform the biased distribution Transformed into an unbiased distribution, the formula is as follows:

[0051]

[0052] Finally, P(T|X) is the final predicted distribution. The implementation of the transformation matrix M is shown below. Figure 3 .

[0053] See Figure 3 ,The update of the conversion matrix of the present invention plays a particularly critical role, which transforms a biased prediction distribution into an unbiased prediction distribution;

[0054] C0: Initialize the matrix first. All values of the matrix are initialized to 1.5.

[0055] C1: Activated with sigmoid nonlinear activation function

[0056] C2: Replace the diagonal elements of the transformation matrix corresponding to the base class with the diagonal elements corresponding to the incremental class, and require that the diagonal elements of the base class are smaller than the diagonal elements of the incremental class. This can correct the distribution biased towards the base class.

[0057] C3: Two-norm normalization, so that all operations on the transformation matrix can be differentiated;

[0058] C4: The biased distribution obtained by adding I3 Substitute the conversion matrix M obtained by summing C1-C3 into Formula 4 to obtain the final desired prediction distribution P(Y|X).

[0059] See Figure 4 In the step 2 of the present invention, the base class model is first trained, and incremental learning is performed according to steps 3 and 4. After the incremental learning is completed, the incremental learning model is used to test the test set of all trained data categories (including the base class and all incremental classes). During the test, the test picture is input, and the features are extracted by the trained incremental class feature extractor, and then the feature extractor is used to extract the features. Calculate the cosine similarity and multiply the obtained distribution by the trained transformation matrix M to get the final predicted distribution.

[0060] Example

[0061] Dataset partitioning module, see Figure 4 This embodiment uses the academic dataset CIFAR-100, which contains 60,000 RGB images in 100 categories, with 500 training data for each category and 100 test data for each category. According to the division of incremental small sample image classification, the first 60 categories are used as base classes, and each category contains 500 training data and 100 test data; subsequently, it is divided into 8 incremental tasks in a 5-way 5-shot manner, specifically, each task contains 5 categories, and each category contains 5 training sets and 100 test sets. The above division requires that the categories of each task cannot be repeated. This embodiment uses an RTX 2080Ti as hardware support.

[0062] Base class training, see Figure 1 The model was trained with training data from 60 categories, with a batch size of 120 images. SGD (stochastic gradient descent) was used for optimization, with an initial learning rate of 0.01, which was decayed by 0.01 at the 60th and 80th iterations, for a total of 100 iterations. The model was optimized using the multi-class cross entropy loss function.

[0063] Incremental task training, see Figure 2 The model input is incremental task data (5-way 5-shot, 5 categories, 5 images per category) and a portion of backtracking data (one image per category, a total of 60 images). The specific implementation steps are as follows:

[0064] 1), I0 performs data enhancement;

[0065] 2) After I1, feature extraction is performed. During feature extraction, in order to better adapt to the feature extraction of new data, the first three layers of the feature extractor Resnet-18 are fixed, and only the last layer is fine-tuned. At the same time, in order to prevent catastrophic forgetting, knowledge distillation is performed using formula (1);

[0066] 3), I2 prototype point merging; first use the base class data to extract the prototype point P of the base category through the base class feature extractor B ={P1, P2, ..., P 60}, the extraction method is shown in formula (2), using the incremental class data to extract the prototype point P of the incremental class through the current task model I ={P 61， P 62 , P 63 , P 64 , P 65}, then merge the prototype points of the base class and the incremental class P = {P1, P2, ..., P 65};

[0067] 4), I3 cosine similarity classification, using cosine similarity as a metric, the prototype point P and the image features obtained in 2) are calculated for cosine similarity to obtain the predicted distribution

[0068] 5), I4 conversion matrix correction, see Figure 3 , initialize a matrix B ij ER 65×65 , first through sigmoid activation w ij For B ij The elements in row i and column j in ; then replace the diagonal elements B ii =1,0≤i≤60,B jj =1.02, 61≤j≤65, and then normalized by the two norm After obtaining the transformation matrix M, the distribution deviation is corrected using the transformation matrix

[0069] This example is tested on CIFAR-100, and the accuracy of each task and the average accuracy are compared with the benchmark method to prove the effectiveness of the present invention. The comparison results are shown in Figure 5 The present invention has improved the overall accuracy (direct fine-tuning-total accuracy vs. conversion matrix-total accuracy). At the same time, due to the effect of distribution deviation correction, the accuracy of the incremental class has been greatly improved (direct fine-tuning-incremental class accuracy vs. conversion matrix-incremental class accuracy).

[0070] See Figure 6The confusion matrix visualizes the effect of bias correction, where the horizontal axis 0-59 represents the base class, 60 and above represent the incremental class, and the vertical axis represents the true label. The darker the diagonal color in the figure, the more correctly the class is predicted. The first row is the result without bias correction, and the second row is the result with bias correction. Each column represents the 2nd, 4th, 6th, and 8th incremental learning respectively. It can be seen that after the bias correction of the present invention, the prediction number of the incremental class is significantly improved, and is no longer biased towards the base class, which proves the effectiveness of this method.

Claims

1. An incremental small sample image classification method based on bias correction, characterized in that: The specific steps are as follows: Step 1: Dataset preparation and preprocessing Collect image data to be classified and divide it into different tasks according to categories. The first task requires a large amount of data and uses all the data for each category. Subsequent incremental tasks are set according to the N-way K-shot method, where N is the number of categories and K is the number of samples in each category. Step 2: Base task model construction and training The base class task model consists of a base class feature extraction network and a base class classifier. The base class feature extraction network uses Resnet-18, the base class classifier uses a fully connected layer, and the base class task model uses a cross-entropy loss optimization model. This step aims to use a large amount of existing labeled data to train a base class feature extraction network. Step 3: Incremental task model construction 3-1: Constructing an incremental feature extraction network: fix the first three layers of the base feature extraction network, and use the last layer to adapt to incremental tasks; 3-2: Initialize the transformation matrix M with a dimension equal to the sum of the number of incremental classes and the number of base classes. Use the sigmoid nonlinear activation function to activate it. Replace the diagonal elements of the transformation matrix corresponding to the base class with the diagonal elements corresponding to the incremental class. The diagonal elements of the base class must be smaller than the diagonal elements of the incremental class. Step 4: Incremental task training 4-1: Extract the image features of base class data using the base class feature extraction network and base class training data base , and calculate the prototype point of each category of the base class, the formula is as follows: in is the prototype point of the cth class of the base class, and K is the number of pictures of each category of the base class; 4-2: Extract incremental image features feat using the incremental feature extraction network built in 3-1 and incremental training data novel ; 4-3: Use the incremental image features extracted in 4-2 to calculate the incremental image prototype point. The formula is as follows: in is the prototype point of the cth class of the incremental class, N is the number of pictures of each category of the incremental class; Merge with the base class prototype point obtained in 4-1, It is the prototype point of all categories and is used for classification; 4-4: Using cosine similarity as a metric, we use the prototype points obtained in 4-3 and the features obtained in 4-2 to classify and obtain a predicted distribution where the incremental class is biased towards the base class. Indicates the probability that the predicted sample X is predicted to be category Y; 4-5: Use the conversion matrix of 3-2 to correct the predicted distribution of 4-4, The corrected distribution P(Y|X) is an unbiased predictive distribution.

Images