A lightweight prototype container-based small sample class incremental learning system

Abstract

The application discloses a kind of light based on prototype container small sample class incremental learning system, including multi-stage pre-training method, the algorithm of calculating prototype and the storage and incremental update method of prototype in prototype container.The method with the strategy of multi-stage pre-training constantly improves the sample representation ability of model feature extraction, and calculates each class prototype as classification standard.Using prototype container stores the prototype of all current classes, only needs to update new class prototype in subsequent incremental process, without further training.The application makes up the problem that classification head network parameter increases infinitely due to the increasing number of classes in the existing small sample class incremental training task and the inconvenience of incremental process in complex scenarios, only by multi-stage pre-training model to calculate class prototype, and prototype container replaces the traditional classifier, optimizes the incremental training process from the algorithm level, so that small sample class incremental learning system is convenient and efficient.

Description

TECHNICAL FIELD

[0001] The application relates to a lightweight small sample class incremental learning system based on a prototype container and belongs to the technical field of image class incremental recognition in pattern recognition. BACKGROUND

[0002] Excellent classification performance often depends on a large amount of labeled data. With the continuous development of society, data is constantly expanding and updating. When new data can be obtained, the old data and the new data can be used to train the model from scratch to build a classification system, ensuring that the model meets the simultaneous recognition of new and old classes. However, with more and more data, training from scratch will consume a large amount of training resources and time, inevitably hindering their practical application. In addition, not all old data can be effectively reused. For example, in a certain species of image recognition, old class samples will be difficult to continue to be used for model training due to loss or damage caused by improper preservation. Moreover, if the number of new classes is limited, the imbalance between new and old classes during training will cause the model to easily overfit and lose new class information. For example, in a more complex data privacy scenario, multiple medical institutions jointly maintain a certain type of disease classification diagnosis, and high-privacy patient data is difficult to integrate into the training of the diagnosis system from scratch. At this time, the image recognition field indeed needs an efficient and lightweight class incremental learning algorithm to face the complex small sample problem and the old class sample loss problem.

[0003] Although the small sample class incremental learning method has achieved ideal results in solving the small sample problem and the old class knowledge forgetting problem in the incremental training process. However, it lacks the lightweight and efficient consideration of complex scenarios, resulting in the problems of unlimited increase of classification head network parameters due to the increase of the number of classes and the occupation of a large amount of storage space by the storage of old sample image data. In order to solve these two big problems, there are two most commonly used technologies. First, the data level method, which improves the ability of the model to extract sample features through image data data enhancement methods. Second, the algorithm level method, which efficiently completes the image recognition task by calculating the target prototype of the pre-trained model and adopting the cosine similarity matching method.

[0004] The present application starts from the algorithm level. Firstly, a multi-stage pre-training method is used to fully utilize a large amount of basic category data to train a pre-training model. Multi-stage pre-training can guide model training at different stages, and the stage-by-stage improvement of the model sample expression ability. Secondly, a prototype container strategy is designed, which calculates the prototype of each category to represent the samples of the category and serves as the standard for classification of the category. This method effectively solves the problem of unlimited increase of classification head network parameters due to the increasing number of categories. Moreover, compared with the strategy of storing original images, storing category prototypes can save more storage space and provide high-dimensional insertable space for future new categories. For the introduction of new categories, the new category prototype is calculated and updated to the prototype container, and this process does not require further model training, shortening the time required for incremental training. The training process of the present application only occurs in the basic categories, making it possible for more complex real-world category incremental learning. The small sample category incremental learning system based on the prototype container proposed in the present application does not require further training and large storage, ensuring the convenience and efficiency of the entire incremental process. SUMMARY

[0005] Technical problem: In view of this, the present application provides an image category incremental recognition system that can solve the problem of unlimited increase of classification head network parameters and limited storage space in category incremental learning. The multi-stage pre-training improves the sample expression ability of the pre-training model. The pre-training model is used to extract the prototype of each category, which serves as the classification standard and can ignore the problem of unlimited increase of classification head network parameters due to the increasing number of categories. By designing a prototype container to store new and old category prototypes, the recognition and classification of new and old categories can be maintained, and the prototype can better alleviate the problem of limited storage space during the incremental process. The present application can better ensure the efficiency and convenience of the category incremental process.

[0006] Technical solution: Firstly, the image data is divided into different sessions, each session containing a training set and a test set. The first session contains basic data, which has more categories and more training samples. The second session starts with a training set that meets the small sample scenario, with only a small number of labeled samples. During the process of accessing the current session, all previously encountered session samples cannot be accessed, but all previously encountered categories need to be classified. Secondly, the pre-training model is trained using the multi-stage pre-training strategy based on the data of the first session. After training, all category prototypes of the first session are extracted and stored in the prototype container, and the prototypes are used as classification standards for recognition testing on the test data of the first session. Then, when the second session data arrives, the pre-training model is used to extract all category prototypes of the second session and add them to the prototype container for updating. After the second session, all previously encountered categories need to be tested simultaneously. Repeat until all sessions are completed, and output the final recognition results of each session.

[0007] The technical solution adopted by this invention to solve its technical problem can be further refined. First, considering that multi-stage pre-trained models need to guide the pre-training of models at different stages, this paper designs a four-stage pre-training strategy. The pre-trained model's ability to represent samples is improved through four stages: cross-entropy classification, contrastive data augmentation, clustering, and relation learning. Since relation learning aims to learn the correlation between samples, this stage serves as a supplement to features and does not contain classification information. Second, the prototype is used as the classification standard, and cosine similarity is calculated to determine the recognition result. The pre-trained model extracts relevant features for each test sample and calculates cosine similarity with all relevant features in the prototype container. The category with the highest similarity score is taken as the recognition result.

[0008] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0009] Existing incremental learning methods with few samples often result in an infinite increase in the parameters of the classification head network due to the continuous addition of categories, leading to high storage costs for the entire model. Furthermore, the incremental process requires storing old samples, which consumes significant storage space. This invention, for the first time, fully leverages the large amount of basic category data in the first session, training the pre-trained model through multi-stage pre-training to continuously improve its ability to represent samples. Secondly, it designs the concept of a prototype container to store prototype samples of both new and old categories, using these as recognition criteria. This strategy keeps the total number of model parameters constant, and the model does not require further training after pre-training. The incremental process is manifested in the continuous updating of the prototype container, making the entire process efficient and lightweight.

[0010] When a new session and a new category arrive, the prototype container needs to be updated. Features of a limited number of new category samples are extracted using a pre-trained model, and category prototypes are calculated from the same category features. These category prototypes are then inserted into the prototype container to complete the update after each new session. This process does not involve the training process, saving training time, and the stored old category prototypes require less storage space compared to the original images, alleviating storage pressure during incremental processes.

[0011] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will become apparent to those skilled in the art upon studying the following description and practicing the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the written description, claims, and drawings. It will be understood by those skilled in the art that the objects and advantages achievable with the invention are not limited to those specifically described above, and that the above and other objects achievable by the invention will become more clearly understood from the following detailed description. Attached Figure Description

[0012] Figure 1is a multi-stage pre-training flowchart of the present application;

[0013] Figure 2 is a prototype container update flowchart of the present application; DETAILED DESCRIPTION

[0014] In order to more clearly describe the content of the present application, the examples described below are not intended to limit the scope covered by the present application. In order to make the purpose, technical solutions and advantages of the present application more clear and apparent, the present application will be further described in detail below in conjunction with the embodiments and drawings. The illustrative embodiments of the present application and their descriptions are used to explain the present application, but are not intended to limit the present application.

[0015] It should be emphasized that, in order to avoid obscuring the present application due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present application are shown in the drawings, and other details not closely related to the present application are omitted. A lightweight prototype container-based small sample class incremental learning system of the present application includes the following steps:

[0016] Step 1: The background divides the image sample data into multiple sessions S according to the specific class incremental problem description. Each session contains corresponding training data D and test data T. In the base session S0, there are more classes and more samples for the model to fully train. In subsequent sessions, the training sample classes are novel and have not been seen before, and the number is limited. In each session, the training data and the test data have common classes L, and the classes are not shared and disjoint between sessions. After each session, all the classes that have appeared need to be recognized at the same time.

[0017] Step 2: Use the base session sample data for multi-stage pre-training, as described in Figure 1 The input data is finally obtained after multi-stage training. The specific steps are as follows:

[0018] Step 2.1: First, initialize the model network weight W and use the base session data D0 to train the first stage model from scratch. The specific steps are as follows:

[0019] Step 2.1.1: In order to meet the basic classification requirements, cross-entropy loss is used to train Model1 in the first stage. First, perform random data augmentation on the data, including random inversion, random rotation, random cropping, etc., and standardize the image data.

[0020] Step 2.1.2: Input the data augmented and standardized image into the neural network to train the classifier C to minimize the cross-entropy loss L1:

[0021] L1(x i , y i, C) = -log(P(y i |x i , C)), (1)

[0022] where x i , y i are input image data and its label respectively, P(y i |x i , C) represents the class probability after softmax layer. After the first stage of training, Model1 has completed the classification task preliminarily, and the output of the layer before the softmax layer has basic expression ability.

[0023] Step 2.2: In order to further improve the feature expression ability of the model, the contrast learning strategy is adopted. Contrast learning focuses on learning the common features between similar examples. Therefore, Model2 is obtained by carrying out the second stage of training on the basis of Model1, and the specific steps are as follows:

[0024] Step 2.2.1: First, for each real data x i , two random data augmentations are performed to obtain and Data augmentation methods include random inversion, random rotation, random cropping, etc. and are regarded as the augmented data of the real data x i , and the maximum similarity of the output features between the original data and the augmented data is ensured after the same data augmentation in this stage.

[0025] Step 2.2.2: x i , and are input into the neural network at the same time, and the contrast loss is minimized:

[0026]

[0027] where cos() represents the cosine similarity, and f() represents the feature output by the model.

[0028] Step 2.2.3: Secondly, it is ensured that the real samples of the same class in a batch ensure the maximum similarity:

[0029] L same (R, Y) = sum((I - R * )·(YY T )), (3)

[0030] where R is a batch of data, Y is the one-hot form of the label matrix in the batch data. R * represents the cosine similarity matrix of R, Let I represent the cosine similarity between the i-th sample and the j-th sample. * For matrices of the same size, all values ​​are 1. The dot (·) indicates element-wise multiplication.

[0031] Step 2.2.4: Finally, supplementary cross-entropy loss L1 based on real data ensures that the learned features are constrained by the class. Therefore, the overall goal of the second stage can be summarized as follows:

[0032] L2 = L a +L same +L1 (4)

[0033] The feature representation capability of Model2 can be further improved through the second stage of training.

[0034] Step 2.3: To improve the matching ability of the prototype, clustering learning is used to ensure that the features of similar samples are very similar, thereby improving the quality of the computed prototype. Model 3 is trained based on Model 2, and the specific steps are as follows:

[0035] Step 2.3.1: First, calculate the prototypes for each category in each round of training:

[0036]

[0037] Where P c The prototype representing category c, S c Let S represent the sample set of category c, and |S| c | represents the number of samples. Through f(x) i ) and P c Find the sample x that is most similar to this class. c .

[0038] Step 2.3.2: Next, take each x in a batch R. i and the corresponding x c This is simultaneously input into the neural network to minimize the clustering loss:

[0039]

[0040] Step 2.3.3: Simultaneously, to avoid losing information from the first and second phases, L1 and L2 were added. same This guides the training of Model 3. Finally, L3 can be summarized as:

[0041] L3 = L p +L1+L same (7)

[0042] Through the third stage of training, the feature extractor of Model3 has stronger sample expression ability, and the prototype calculated will be more accurate.

[0043] Step 2.4: Since class-incremental learning needs to be solved in a small sample scenario, the model needs to have the ability to extract features from a small number of samples. Therefore, in this stage, Model4 is trained using a meta-learning strategy. If the previous classification information is included in the meta-learning process, it will affect the training of meta-features. Therefore, this stage is trained from scratch, and the meta-features will be represented as complementary features. The specific steps are as follows:

[0044] Step 2.4.1: First, each meta-task belongs to the N-way K-shot setting, including N*K support samples and N*K query samples, and the prototype P is constructed using the support set by formula (5) c .

[0045] Step 2.4.2: Second, calculate the average loss between a batch of meta-tasks:

[0046]

[0047] where |T| represents the number of meta-tasks in the batch, represents the query set of the nth task in the batch, P n represents the class prototype in the nth task in a batch. P(y i |x i , P n ) represents the probability that the query sample x i belongs to the y i class in the meta-training network, which is expanded as follows:

[0048]

[0049] where represents the cth class prototype of the nth task. τ is the relaxation hyperparameter in the prototype network.

[0050] Step 2.5: Through these four stages, the expression of the sample will be completed by Model3 and Model4 respectively. The basic feature extractor of Model3 has rich image classification feature information, while the meta-feature extractor of Model4 is more suitable for effective feature extraction in a small sample scenario.

[0051] Step 3: Calculate the target prototype and store it in the prototype container. After obtaining Model3 and Model4, calculate the classification prototype and meta-prototype of the basic session class using formula (5) and store them in the prototype container. After multi-stage pre-training, the basic session classes are obviously clustered together in the feature space, which makes it possible to insert new class prototypes.

[0052] Step 4: New incremental session flows in, compute new class prototypes and container update. Whenever a new session is introduced, as described in Figure 2 the new class prototype is computed by limited new class samples and updated into the prototype container. This process does not require further training, saving incremental training time.

[0053] Step 5: Test samples compute relevant features to get final prediction results. Test samples are respectively passed through Model3 and Model4 to obtain relevant features. And calculate the cosine similarity with all class prototypes in the prototype container. The similarity score Score base is calculated by the base classification prototype meta Weighted fusion. The final similarity score Score is:

[0054] Score = ω * Score base + (1-ω) * Score meta , (10)

[0055] Where ω is a hyperparameter, ranging from [0, 1]. The class with the highest similarity score will be the final discriminant result of the test sample.

[0056] The specific embodiments of the application are described above with reference to the accompanying drawings. However, those skilled in the art can understand that the components, systems and methods described in conjunction with the embodiments disclosed herein can be realized in hardware, software or a combination of both, and that several improvements and equivalent replacements can be made without departing from the spirit and principles of the application. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application. The improved and equivalent replaced technologies and solutions of the claims of the present application all fall within the protection scope of the present application.

[0057] Experimental design

[0058] Selection of experimental data set: The real image data set used in this paper is selected from CIFAR-100, miniImageNet and CUB200 standard data. Their detailed information is shown in the following table.

[0059] Dataset Sample size Number of classes Total number of samples CIFAR-100 32*32*3 100 60000 miniImageNet 84*84*3 100 60000 CUB200 224*224*3 200 6000

[0060] All datasets adopt the data division strategy commonly used in small sample class incremental tasks. For CIFAR-100 and miniImageNet, 100 classes are divided into 60 base classes and 40 incremental classes. The 40 incremental classes are further divided into 8 new sessions. Each session has 5 new classes, and the 5-Way 5-Shot setting is adopted, that is, there are only 5 samples for each class in each new session. For CUB200, 200 classes are divided into 100 base classes and 100 incremental classes. The 100 incremental classes are further divided into 10 new sessions. Each session has 10 new classes, and the 10-Way 5-Shot setting is adopted, that is, there are only 5 samples for each class in each new session. After each session carrying new classes, the classification task of the test samples of all classes appearing in the last session needs to be completed. The performance evaluation index after each session is the accuracy of the test samples of all classes that have appeared.

[0061] Comparative model: the system proposed in the application is named PC-FSCIL. The classification performance is compared among SPPR, DC, DDEMD and CEC.

[0062] Performance measurement method: the accuracy (Accuracy) of the multi-class standard dataset used in the experiment is used for evaluation. It shows the number of correctly predicted samples accounts for the total number of samples. The definition of Accuracy is as follows:

[0063]

[0064] Experimental results:

[0065]

[0066]

[0067] The data in the table are the prediction results under the Accuracy index, and each row of the table corresponds to a method and each column corresponds to a session. The best results are marked in bold.

[0068] According to the classification results of various algorithms, the example of the application has obvious advantages in the three datasets, and the comprehensive recognition rate can reach the highest. In addition, no further training is required in the incremental process, which saves the training time and reduces the storage pressure of the entire incremental system.

Claims

1. A lightweight prototype container-based few-shot class incremental learning system, characterized in that: The method comprises the following steps: 1) The background divides the image sample data into training data and test data according to a specific federal problem description; 2) Pre-training model is obtained by using basic conversation sample data for multi-stage pre-training, and the specific process is as follows: first, the model network weight W is initialized, and the first stage model is trained from scratch using the basic conversation data D0; second, in order to further improve the feature expression ability of the model, a contrast learning strategy is adopted, which focuses on learning the common features between similar examples, and the specific steps include: first, for each real data x i Two random data augmentations are performed And The data augmentation methods include random inversion, random rotation, random cropping, And The augmented data is regarded as the real data x i , which guarantees the maximum similarity of the output features after the same data augmentation and the original data in this stage; second, x i , And are input into the neural network at the same time, and the contrast loss is minimized: Wherein cos() represents the cosine similarity, f() represents the feature output by the model; Finally, the similarity of real samples of the same category in a batch is guaranteed to be maximum: L same (R,Y) = sum((I - R * ) · (YY T )), (2) where R is a batch of data, Y is the one-hot form of the label matrix in the batch data; R * represents the cosine similarity matrix of R, represents the cosine similarity between the i-th sample and the j-th sample; I is a matrix with the same size as R * all values are 1; · represents the corresponding multiplication operation of the matrix elements; Then, in order to improve the matching ability of the prototype, clustering learning is used to ensure that the features of similar samples are very close, so as to improve the quality of calculating the prototype. The specific steps include: first, calculate the prototype of each category in each round of training: where P c represents the prototype of class c, S c represents the sample set of class c, and |S c | represents the number of samples; find the most similar sample x i to this class by f(x c ) and P c ; second, input each x i in a batch R and the corresponding x c to the neural network as input at the same time, minimize the clustering loss, Finally, since class-incremental learning needs to be solved in a small sample scene, the model needs to have the ability to extract features from a small number of samples, and the specific steps include: first, each meta-task belongs to the N-way K-shot setting, including N*K support samples and N*K query samples, and the prototype P is constructed by using the support set according to formula (3) c ; second, the average loss between a batch of meta-tasks is calculated: where |T| denotes the number of meta-tasks in a batch, denotes the query set of the n-th task in a batch, P n denotes the class prototype in the n-th task in a batch; P(y i | x i , P n ) denotes the probability that the query sample x i belongs to the y i -th class in the meta-training network, which is expanded as: wherein denotes the c-th prototype of the n-th task; τ is a relaxation hyperparameter in the prototype network; 3) Calculate the target prototype and store it in the prototype container; 4) New incremental session flows in, calculate new category prototype and container update; 5) Calculate the relevant features of the test sample to get the final prediction result.

2. The lightweight prototype container-based small sample class incremental learning system according to claim 1, wherein: Step 1) The background divides the image sample data into training data and test data according to a specific federal problem description; The specific steps include: dividing the image sample data into multiple sessions; Each session contains corresponding training data and test data; In the basic session, there are more categories and more samples for the model to fully train; In the subsequent session, the training sample category is novel and has not been seen before, and the number is limited; In each session, the training data and test data have common categories, and the categories are not shared and disjointed between sessions; After each session, all categories that have appeared need to be recognized at the same time.

3. The lightweight prototype container-based small sample class incremental learning system according to claim 1, wherein: Initialize the model network weight W to train the first stage model from scratch using the basic session data D0; The specific steps include: in order to meet the basic classification requirements, cross-entropy loss is used for training in the first stage, first, random data augmentation is performed on the data, including random inversion, random rotation, random cropping, and standardization operation on the image data; Finally, the image after data augmentation and standardization is input into the neural network to train the classifier to minimize the cross-entropy loss: L1(x i ,y i ,C) = -log(P(y i |x i ,C)), (7) where x i ,y i are the input image data and its label respectively, P(y i |x i , C) represents the class probability after the softmax layer; after the first stage of training, Model 1 has completed the classification task preliminarily, and the output of the layer before the softmax layer has basic expression ability.

4. The lightweight prototype container-based small sample class incremental learning system according to claim 1, wherein: Step 3) Calculate the target prototype and store it in the prototype container; The specific steps include: calculating the target prototype and storing it in the prototype container; After obtaining Model3 and Model4, the classification prototype and meta prototype of the basic session category are calculated using formula (3), and the classification prototype and meta prototype are stored in the prototype container.

5. The lightweight prototype container-based small sample class incremental learning system according to claim 1, wherein: Step 4) New incremental session flows in, calculate new category prototype and container update; The specific steps include: whenever a new session is introduced, calculate the new category prototype through the limited new category samples, and update it to the prototype container; This process does not require further training, saving incremental training time.

6. The lightweight prototype container-based small sample class incremental learning system according to claim 1, wherein: The test sample of step 5) is calculated to obtain the final prediction result; the specific steps include: obtaining the relevant features of the test sample through Model 3 and Model 4 respectively; and calculating the cosine similarity with all category prototypes in the original container; calculating the similarity score of the basic classification prototype base The similarity score calculated with the meta prototype meta Weighted fusion; the final similarity score Score is: Score = ω * Score base + (1 - ω) * Score meta , (8) Wherein ω is a hyperparameter, ranging from [0, 1]; The category with the maximum similarity score will be the final discrimination result of the test sample.

Images