Image classification model based on class incremental learning and training method and classification method thereof

Abstract

The invention provides an image classification model based on class incremental learning and a training method and a classification method thereof. The classification model includes a solver and a feature generator. Wherein the solver comprises a feature extractor, a classifier and a plurality of auxiliary classifiers. And the feature extractor is used for extracting features according to the input tasks and outputting final features. And the classifier is used for classifying the tasks according to the final features. And the auxiliary classifier is used for generating embedded knowledge of the embedded distillation according to the output of the learning module during model training. And the feature generator is used for generating generated features of the historical tasks under the guidance of the classifier, and training the solver by performing loss fitting on the generated features and the current task features. According to the invention, richer knowledge embedded in a network intermediate block can be migrated from an old frozen feature extractor to a current feature extractor to prevent the feature extractor from being forgotten, and historical task data or prototypes do not need to be stored.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image classification model based on class incremental learning and a training method and a classification method thereof. Background Art

[0002] As a disruptive technology in the field of computer vision, deep learning is widely used in the field of image processing, and has achieved remarkable success in civilian fields such as face recognition, autonomous driving, and medical image processing. In dynamic scenes, deep learning models need to be able to handle tasks that arrive sequentially. However, traditional model training methods usually lead to catastrophic forgetting problems, that is, once a new data set is used to train an existing model, the model will lose its ability to recognize the original data. In order to overcome catastrophic forgetting, the model must, on the one hand, demonstrate the ability to integrate new knowledge from new data and refine existing knowledge (plasticity), and on the other hand, it must prevent new inputs from significantly interfering with existing knowledge (stability).

[0003] In order to improve the plasticity of the model while taking into account stability, researchers have proposed a class incremental learning method. The currently widely used class incremental learning method requires storing examples (representative samples of historical tasks) to incrementally train feature extractors, or storing prototypes (representative features of historical tasks) to estimate incremental feature distributions. However, storing examples violates the principle of user data privacy protection, and fixed prototypes may not be consistent with the incrementally changing feature distribution, which will seriously hinder the application of class incremental learning methods for image classification in realistic dynamic scenes. For example, in existing technologies, class incremental learning using knowledge distillation technology transfers the probabilistic knowledge between prediction layers from the old frozen network to the current trainable network. The deep learning network includes two parts, the feature extractor and the classifier, which correspond to the incremental learning method based on the feature extractor and the incremental learning method for prototype estimation, respectively. However, the method based on the feature extractor needs to store some historical examples, and the method for prototype estimation needs to store the prototypes of historical classes.

[0004] Therefore, there is an urgent need for an image classification model based on class incremental learning without storing historical examples or prototypes to meet the application requirements of incremental learning for privacy protection. Summary of the invention

[0005] In order to solve the above problems, the present invention provides an image classification model based on class incremental learning and a training method and a classification method thereof.

[0006] According to a first aspect, an embodiment of the present invention provides an image classification model based on class incremental learning, comprising a solver and a feature generator; wherein the solver comprises a feature extractor, a classifier and a plurality of auxiliary classifiers; the feature extractor is used to perform feature extraction according to an input task and output a final feature, comprising a plurality of learning modules for feature extraction connected in sequence, the output of a previous learning module being the input of a next learning module; the classifier is used to classify the task according to the final feature; the auxiliary classifier, the auxiliary classifier is one-to-one connected to the outputs of other learning modules except the last learning module, and is used to generate embedded knowledge for embedding distillation according to the outputs of the learning modules during model training; and the feature generator is used to generate generated features of historical tasks under the guidance of the classifier, and the generated features are fitted with the current task features through loss to train the solver.

[0007] Preferably, the feature extractor and classifier adopt the network structure of convolutional neural network ResNet-18; the feature generator includes a variational encoder.

[0008] According to a second aspect, an embodiment of the present invention provides a method for training the image classification model based on class incremental learning described in the first aspect, comprising an initial stage training and an incremental training stage; wherein the initial stage training comprises: training the initial solver of the image classification model according to the original task obtained in the initial stage until convergence; freezing the initial feature extractor and the initial classifier in the converged solver, and training the initial feature generator according to the original task in the initial stage until convergence; each incremental training stage comprises: initializing the image classification model trained in this stage based on the parameters of the image classification model trained in the previous stage The parameters of the class model are obtained; the data of the original task input in this stage is input into the feature generator trained in the previous stage after extracting features through the feature extractor obtained in the previous stage to obtain the generated features of the historical original task; according to the generated features of the historical original task and the original task obtained in this stage, the solver in the image classification model initialized in this stage is trained until convergence; the initial feature extractor and the initial classifier in the converged solver are frozen; according to the solver trained for the original task of this stage and the feature generator trained in the previous stage, the feature generator in the image classification model initialized in this stage is trained until convergence.

[0009] Preferably, in the initial stage training, the initial solver of the image classification model is trained until convergence according to the original task obtained in the initial stage, including: training the initial feature extractor and the initial classifier according to the original task obtained in the initial stage, and calculating the first loss function; performing enhancement processing according to the original task in the initial stage to obtain the enhanced task in the initial stage; training the initial feature extractor and the initial auxiliary classifier according to the enhanced task in the initial stage, and calculating the second loss function; taking the sum of the first loss function and the second loss function as the third loss function, and based on the third loss function, updating the parameters of the initial feature extractor, the initial classifier and the initial auxiliary classifier until the initial solver converges.

[0010] Preferably, in the initial stage training, the initial feature extractor and the initial classifier in the solver that have converged are frozen, and the initial feature generator is trained until convergence according to the original task of the initial stage, which includes: freezing the initial feature extractor and the initial classifier in the solver that have converged, inputting the original task of the initial stage into the initial feature generator to obtain the generated features of the original task of the initial stage; calculating the fourth loss function of the initial stage based on the final features output by the initial feature extractor, the generated features of the original task of the initial stage, the parameters of the frozen initial classifier, and the noise parameters; and updating the parameters of the initial feature generator based on the fourth loss function of the initial stage until the initial feature generator converges.

[0011] Preferably, in each incremental stage of training, the parameters of the image classification model trained in the previous stage are used to initialize the parameters of the image classification model trained in this stage, including: initializing the solver trained in this stage with the parameters of the solver trained in the previous stage; initializing the feature generator trained in this stage with the parameters of the feature generator trained in the previous stage.

[0012] Preferably, in each incremental stage training, according to the generated features of the historical original task and the original task obtained in this stage, the solver in the image classification model after initialization in this stage is trained to convergence, including: inputting the original task obtained in this stage into the initialized solver and the solver trained in the previous stage, and calculating the fifth loss function according to the output of the feature extractor in the initialized solver and the output of the feature extractor in the solver trained in the previous stage; performing enhancement processing according to the original task obtained in this stage to obtain the enhanced task of this stage; inputting the enhanced task of this stage into the initialized solver and the solver trained in the previous stage, and The sixth loss function is calculated according to the output of the auxiliary classifier in the initialized solver and the output of the auxiliary classifier in the solver after training in the previous stage; the features extracted by the feature extractor in this stage are input into the feature generator after training in the previous stage to obtain the generated features of the historical original task; the initialized solver is trained based on the generated features of the historical original task and the original task in this stage to calculate the seventh loss function; the weighted sum of the fifth loss function, the sixth loss function and the seventh loss function is used as the eighth loss function, and based on the eighth loss function, the parameters of the initialized solver are updated until the initialized solver converges.

[0013] Preferably, in each incremental stage training, according to the solver trained for the original task of the present stage and the feature generator trained in the previous stage, the feature generator in the image classification model initialized in the present stage is trained until convergence, including: initializing the feature generator of the present stage according to the feature generator trained in the previous stage; inputting the original task of the present stage into the initialized feature generator, and calculating the fourth loss function of the incremental stage; after inputting the original task of the present stage into the solver trained for the original task of the present stage, its output is then input into the initialized feature generator to calculate the ninth loss function; taking the weighted sum of the fourth loss function of the incremental stage and the ninth loss function as the tenth loss function, and updating the parameters of the initialized feature generator based on the tenth loss function until the initialized feature generator converges.

[0014] Preferably, the original task is enhanced to obtain an enhanced task, including: the original task includes multiple images; the images are rotated and / or mirrored to obtain processed images; the processed images constitute the enhanced task.

[0015] According to a third aspect, an embodiment of the present invention provides an image classification method, wherein the method comprises: obtaining an image to be processed; and using an image classification model trained by any method described in the second aspect to process the image to be processed to obtain a classification result.

[0016] Compared with the prior art, the advantages of the present invention are:

[0017] The present invention can migrate the richer knowledge embedded in the middle block of the network from the old frozen feature extractor to the current feature extractor to prevent the feature extractor from forgetting, without storing historical task data or prototypes. The present invention makes full use of the trained classifier, so that the lightweight generator can generate dynamic task features that meet the incremental training. At the same time, the screenshot of the present invention achieves higher accuracy, making incremental learning technology for real-world applications possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0019] Figure 1 A schematic diagram of the workflow of a feature extractor in different training stages of an image classification model based on class incremental learning according to an embodiment of the present invention;

[0020] Figure 2a A schematic diagram of a training process of a solver at different training stages in an image classification model based on class incremental learning according to an embodiment of the present invention;

[0021] Figure 2b A schematic diagram of a training process of a feature training solver generated by a feature generator in an image classification model based on class incremental learning according to an embodiment of the present invention;

[0022] Figure 3 Schematic diagram of a method flow for training an image classification model based on class incremental learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail by specific embodiments below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0024] The inventors found in the process of studying incremental learning that for real-world applications with strict requirements for user privacy protection, it is possible to consider designing an incremental learning model that does not require the storage of historical examples. However, because the number of parameters of the feature extractor is much larger than that of the classifier, it is difficult to transfer sufficient feature extractor knowledge using conventional end-to-end knowledge distillation methods to achieve the purpose of incremental training of feature extractors; on the contrary, the richer historical knowledge embedded in the middle layer of the feature extractor will be lost in the incremental learning process. Moreover, the prior art only saves one historical category feature prototype, which makes it difficult to achieve the overall historical feature distribution estimation.

[0025] Therefore, the inventors proposed a quasi-incremental learning image classification model based on embedding distillation and task-oriented feature generation and a method for training the model, which migrates the rich knowledge embedded in the middle layer of the network from the historically frozen feature extractor to the currently trained feature extractor to prevent the feature extractor from forgetting the knowledge of the historical task. At the same time, a task-oriented feature generation strategy is designed, that is, pseudo features that are as close as possible to the historical task features are generated in the new task based on the features obtained from the historical task, making full use of the trained classifier so that the lightweight feature generator can fit and generate historical features well. In addition, self-supervised auxiliary tasks can be constructed based on the current task to further strengthen the learning of the current task.

[0026] In order to better illustrate the technical solution of the present invention, the following is a detailed description from four aspects: model structure, training samples, embedding distillation and training model in combination with specific embodiments.

[0027] 1. Model Structure

[0028] like Figure 1 As shown, according to one embodiment of the present invention, a schematic diagram of the workflow of a feature extractor in different training stages of an image classification model based on incremental learning is proposed. Specifically, the red part in the figure represents the training data 101 used by the original task in the previous stage and the feature extractor 102 in the model trained in the previous stage (the parameters of the feature extractor are frozen in this stage), and the blue part represents the training data 103 used by the original task in this stage and the feature extractor 104 in the model being trained in this stage. Among them, the parameters of the model 102 trained in the previous stage are fixed and guide the model 104 being trained in this stage 105, and the features extracted by the model in the previous stage are generated in this stage and participate in the model training 106 in this stage. The detailed training process of the classification model of an embodiment of the present invention is shown in FIG. Figure 2a-2b shown.

[0029] like Figure 2aAs shown, the model proposed in the embodiment of the present invention includes a solver and a feature generator 202. Among them, the solver includes a feature extractor 2011, a classifier 2012 and an auxiliary classifier 2013. In the embodiment of the present invention, the network structure of the ResNet-18 deep neural network is used as the module structure of the feature extractor and the classifier. The feature extractor 2011 is used to extract features from the input task image. ResNet-18 includes structures such as multi-layer convolutional layers and pooling layers. By performing convolution operations on the input image step by step, the low-dimensional features of the image are finally obtained to achieve the effect of feature extraction. Therefore, this part of the network structure is selected as the feature extractor. The feature extractor 2011 includes a plurality of learning modules Block 1-Block L for feature extraction that are connected in sequence. These modules correspond to the layer structures in ResNet-18 in sequence. The output of the previous learning module is the input of the next learning module, and the image features are finally obtained. During the training process of the model, the parameters of the feature extractor 2011 in different training stages will be different. After the training of the previous stage is completed, a copy of the feature extractor of the previous stage will be copied and frozen as the feature extractor for guiding the task training in this training stage. After guiding a task, the copy can be discarded. After the training of this stage is completed, the feature extractor trained for the current task is copied again and frozen for training to guide the next training stage. In order to distinguish the feature extractors 2011 in different training stages, F is used here. t-1 represents the feature extractor in the t-1th training stage, which always remains frozen in the tth training stage task, that is, the parameters are kept fixed in the tth training stage task; F t represents the feature extractor trained in the tth training stage, F t In the initialization phase of the tth training phase and the frozen state F t-1 The structure and parameters of are the same. In the training task of the tth training stage, F t-1 Keeping the parameters and structure unchanged, F t The structure of remains unchanged but the parameters are updated according to the training task of the tth training stage. Classifier 2012 is used to classify the task image according to the extracted features. The embodiment of the present invention uses the linear fully connected layer and nonlinear activation layer after the ResNet-18 feature extraction layer as the classifier. The image is processed by F t After the features are obtained, they are input into the classifier 2012 for linear and nonlinear transformations, and finally the classification results are obtained. For other learning modules except the last learning module Block L, the output of the previous learning module is not only input into the next learning module, but also input into an auxiliary classifier 2013. Please refer to Figure 2a , with F t-1 Taking the first learning module (convolutional layer) Block 1 as an example, its output In addition to passing in the next learning module Block 2, it also passes in the same t-1 The first auxiliary classifier connected 2013 (i.e. ). Auxiliary classifier Perform linear and nonlinear transformations on the input to obtain the classification result. t The output of the first learning module In addition to passing in the next learning module, it also passes in the t The first auxiliary classifier connected 2013 (i.e. ). The input is transformed linearly and nonlinearly to obtain the classification result, and the two results are compared to perform embedding distillation. Therefore, the role of the auxiliary classifier 2013 is to perform embedding distillation on the results between modules during training to enhance the stability of the model. After the model is trained, the auxiliary classifier 2013 can be discarded in the application reasoning process, thereby effectively reducing the model deployment storage. The embedding distillation process will be described in detail later.

[0030] like Figure 2b As shown in FIG. 1 , in the process of model training, in order to prevent the model from forgetting the historical tasks that have been learned, it is also necessary to train the feature generator 202. As in the case of the feature extractor, the parameters of the feature generator 202 at different training stages will be different. In order to distinguish the feature generator 202 at different training stages, G is used here. t-1 Denotes the feature generator in the t-1th training stage, G t represents the feature generator trained in the tth training stage, G t In the initialization phase of the tth training phase, G is trained in the previous phase. t-1 The structure and parameters of G are the same. In the training task of the tth training stage, G t The structure of F remains unchanged, but the parameters are updated according to the training process of the tth training stage. After completing the learning of a task, the current F t and classifier 2012, and in the classifier 2012 and the feature generator G of the previous stage (i.e., the t-1 training stage) t-1 The feature generator G of this stage (i.e., the tth training stage) is trained under the guidance of t The feature generator 202 includes an encoder and a decoder, which are used to generate historical task features, and continue to train the solver by fitting the generated historical task features with the current task features through loss; preferably, a variational autoencoder (VAE) is used as the feature generator of the embodiment of the present invention.

[0031] The embodiments of the present invention can migrate the richer knowledge embedded in the middle block of the network from the old frozen feature extractor to the current feature extractor to prevent the feature extractor from forgetting, and make full use of the trained classifier to enable the lightweight feature generator to generate dynamic task features that meet the requirements of incremental training.

[0032] 2. Training Samples

[0033] According to one embodiment of the present invention, the training samples (i.e., the original tasks) used when training the solver and the feature generator are composed of images and the image categories, with the images as input and the image categories as labels. When training the auxiliary classifier, the enhanced training samples (i.e., the enhanced tasks) are used. The enhanced processing may include image rotation processing and / or mirroring processing, for example, the image may be rotated 180 degrees or horizontally mirrored or vertically mirrored to obtain twice the number of inputs of the original task. Preferably, the image is rotated 4 times, each time by 90 degrees, so that the number of input image categories is increased to four times the original number; the rotated image is still in the horizontal and vertical directions, which does not affect the use of other enhanced processing. At the same time, during training, the model can extract the rotation invariant features between images in the enhanced task according to the enhanced task, and maintain the stable learning ability of the model on the historical tasks.

[0034] 3. Embedded Distillation

[0035] Embedding distillation includes the construction of embedded knowledge and the distillation of embedded knowledge. Embedding knowledge construction enables the feature extractor to learn the current task plastically, while embedding knowledge distillation enables the feature extractor to stably remember the learned tasks.

[0036] Please refer to Figure 2a-2b According to one embodiment of the present invention, the embedded knowledge includes: t The final feature f of the output L and the embedded features input to (L-1) auxiliary classifiers 2013 respectively The embedded features are the outputs of all learning modules except the last learning module Block L, where L is the number of learning modules.

[0037] (1) Construction of embedded knowledge

[0038] First, the construction of the final feature requires a trained feature extractor to extract the final feature of the task from the input task. Take the t-1th training stage as an example: Using the task (t-1) obtained in this training stage, the solver S of the t-1th training stage is trained based on the cross entropy loss function shown in formula 1). t-1 , update the parameters of feature extractor 2011 and classifier 2012; in this training, the output of feature extractor 2011 is the final feature of this training stage.t-1 =F t-1 +C t-1 , C t-1 is the classifier in the t-1th training stage.

[0039]

[0040] in, Represents the cross entropy loss function corresponding to the final output feature; is the image set input in task (t-1); is the label set in task (t-1); image set The image x corresponds to the label set The label y in represents the cross entropy loss; represents the expectation on the data of task (t-1); F t-1 (x) represents the feature extractor F that learns task (t-1) in the t-1th training stage t-1 The corresponding function mapping output, C t-1 (·) represents the classifier C that learns task (t-1) in the t-1th training stage t-1 The corresponding function mapping output, Represents composite function cascade, that is, F t-1 The output of (·) is C t-1 The input of (·) is cascaded and compounded.

[0041] Since the embedded features are output by the learning modules other than the last one, in order to obtain (L-1) embedded features, it is necessary to train the auxiliary classifiers with the enhancement task until convergence. The (L-1) auxiliary classifiers 2013 (i.e. The output of ) is the embedded feature of this training stage. It should be noted that in this training process, only the parameters of the feature extractor 2011 and the auxiliary classifier 2013 are updated. Take the t-1th training stage as an example: input the enhanced task obtained from task (t-1), and train F according to the loss function shown in formula 2). t-1 and

[0042]

[0043] in, represents the cross entropy loss corresponding to the output of the auxiliary classifier; σ is the sigmoid function; Represents the function mapping output corresponding to the auxiliary classifier; represents the feature output by the lth learning module during the i-th rotation of the image in the enhancement task of the learning task (t-1); ri represents the i-th rotation process (i.e., rotation of 0°, 90°, 180°, 270°), Represents the image after r i The label corresponding to the rotated image, N represents the number of transformations in the enhancement process.

[0044] (2) Embedding Knowledge Distillation

[0045] When a new task arrives, the embedding knowledge distillation method is used to retain the knowledge of the previously trained feature extractor. Based on the previous content, the embedding knowledge distillation is described in detail using the tth training stage as an example.

[0046] First, in order to make F t Inheritance F t-1 The learned knowledge is used to make the features output by the two feature extractors completely consistent by distillation. t-1 Initialize the parameters of F t ; Input the task t obtained in the tth training stage into the initialized F t and F t-1 ; In order to ensure the effect of model training and prevent feature drift, the L2 norm, which is more sensitive to abnormal data, is selected to construct the loss function shown in Formula 3) F t Train until convergence, update F t Parameters of . Among them, The loss is a loss function that uses the L2 norm (Euclidean distance) to evaluate the difference between the predicted value and the true value.

[0047]

[0048] Among them, X t is the image set input in task t; image Belongs to image set X t ; F represents the learning task t t-1 The corresponding function mapping output is used as the true value; F represents the learning task t t The corresponding function mapping output is used as the predicted value; Indicates that in the image set X t Expectations based on the data.

[0049] Then, the auxiliary classifiers of the model after the t-1th training stage are used Parameters of each auxiliary classifier in the tth training stage are initialized Train F according to the augmentation task of task t t and F tThe auxiliary classifier connected after the l-th learning module Using Kullback-Leibler divergence to construct and embed features The related embedding distillation loss is shown in Equation 4):

[0050]

[0051] in, represents the distillation loss between auxiliary classifiers among blocks, and τ is a parameter related to distillation. represents the KL divergence, Indicates F t-1 The features output by the lth learning module during the i-th rotation of the image in the augmentation task of learning task t; Indicates F t The features output by the lth learning module during the i-th rotation of the image in the enhancement task of learning task t; Indicates F t-1 The auxiliary classifier connected after the l-th learning module The corresponding function mapping output; express The corresponding function mapping output.

[0052] Finally, using Train the solver S for the tth training stage t =F t +C t , C t is the classifier of the t-th training stage, and is used together with the distillation loss to make the solver more flexible while ensuring stability, so as to better ensure that the first t-1 tasks are not forgotten after learning the newly acquired task t. t Stably retain the learned historical task knowledge and use the feature generator 202 (i.e., G t-1 ) and History tab Generate the generated features of t-1 historical tasks Then the generated historical features and current task features are fitted according to the loss shown in formula 5):

[0053]

[0054] Among them, λ CE To balance the weights, its empirical value can be set to the ratio of the number of categories contained in the previous t-1 historical tasks to the number of categories contained in the current task t; w ：t-1 Represents the classifier parameters after training with the first t-1 historical tasks; Indicates in the History tab Generate features from the previous t-1 historical tasks Seek hope.

[0055] 4. Training Model

[0056] Combination Figure 2a , Figure 2b as well as Figure 3 According to one embodiment of the present invention, a training method for an image classification model based on class incremental learning is proposed, including: when learning the initial task, the model has no historical knowledge to remember, and only needs to train the initial solver and the initial feature generator; when a new learning task appears, the solver and the feature generator are incrementally trained.

[0057] In order to prevent the classifier from forgetting during the continuous learning process, it is necessary to train the feature generator 202 (i.e., G t ) to estimate the conditional probability distribution of the first t task features, and then use Gaussian noise stimulation G t generates features based on historical tasks to help subsequent task training. For the current task, a task-oriented generation strategy is proposed, which allows G t Generate more suitable C t The characteristics of the demand, the loss function based on this strategy is shown in formula 6),

[0058] In order to prevent forgetting during the continuous learning process of the classifier, a task-oriented generation strategy is proposed for the current task. This strategy allows the feature generator to generate features that are more suitable for the classifier requirements. The fourth loss function based on this strategy is shown in Formula 6).

[0059]

[0060] in, Represents the generation of G for task t t The corresponding loss, φ t It's G t The encoder parameters, θ t It's G t Decoder parameters. z is Gaussian noise, and p(z) satisfies the standard normal distribution with a mean of 0 and a standard deviation of 1; represents the conditional probability distribution of the decoder output features; Represents the conditional probability distribution of the encoder output features; is the label set in the first t historical tasks (including task t); w t represents the classifier parameters after training with the first t historical tasks; Indicates F t The final features output after learning task t; Indicates the use of G tGenerate the generated features of the first t historical tasks.

[0061] Preferably, in the first training phase of the solver (ie, t=1):

[0062] Input the first task to the initial solver S, use the first task and the enhanced task obtained from the task, and train the initial feature extractor F0, the initial classifier C0 and the initial auxiliary classifier based on the loss function shown in Formula 1) (i.e., the first loss function), Formula 2) (i.e., the second loss function) and Formula 7) (i.e., the third loss function) To converge.

[0063]

[0064] The first training phase of the feature generator (i.e. t=1):

[0065] Input the first task to the initial feature generator G0, and use the first task based on the loss function shown in Formula 6) and Formula 8) (that is, Formula 6) and Formula 8) to obtain the fourth loss function of the initial stage) to train the initial feature generator G0 until convergence.

[0066]

[0067] In the second and subsequent training stages (i.e. t>1, also called incremental training stage):

[0068] When the task t of this stage arrives, the embedded knowledge distillation method is used to retain the knowledge of the previously trained feature extractor. First, in order to make F t , C t , Inheriting the knowledge learned in the previous training stage, the distillation method is used to make the features output by the two solvers completely consistent.

[0069] Preferably, F t-1 Initialize the parameters of F t , using task t to input into the initialized F t and F t-1 , training F t ; Use the parameters of each auxiliary classifier of the model after the previous training stage to initialize the corresponding auxiliary classifier of this training stage, and train F according to the enhanced task of task t t and In order to make C t Stably retain the learned historical task knowledge, using G t-1 With History Tags Generate features for t-1 historical tasks Combine the features of task t to train F t and C tTo converge; in the above process, combined with formula 3) (i.e., the fifth loss function), formula 4) (i.e., the sixth loss function), formula 5) (i.e., the seventh loss function), the loss function shown in formula 9) (i.e., the eighth loss function) is obtained to update F t , C t , Parameters:

[0070]

[0071] In order to prevent the generative model from forgetting the (t-1) tasks that have been learned, a knowledge reconstruction strategy is used to make G t Ability to reconstruct the G t-1 Input the previous t-1 historical tasks to G t , G t In the process, the knowledge reconstruction loss (i.e., the ninth loss function) shown in formula 10) is added to formula 6) to obtain G t The loss function is as shown in Formula 11) (i.e., the tenth loss function).

[0072]

[0073] in, Represents the first t-1 tasks G t The corresponding loss. U{·} represents a discrete uniform distribution, y ：t-1 Represents the number of all categories contained in the previously learned (t-1) tasks. t-1 It's G t-1 The decoder parameters of λ and λ VAE It is the ratio of the number of categories contained in the previous t-1 tasks to the number of categories contained in the current task t.

[0074] The embodiment of the present invention can migrate the richer knowledge embedded in the middle block of the network from the old frozen feature extractor to the current feature extractor to prevent the feature extractor from forgetting. During the training process, the data (samples) of the historical tasks are not stored. Only the feature extractor trained by the historical tasks is used to guide the training and is discarded after the current task is trained. Because the feature generator is trained in the embodiment of the present invention, Gaussian noise can be input and combined with the current task to extract features to directly generate pseudo features that are as close to the historical sample features as possible, so there is no need to retain the prototype of the historical features to generate pseudo features that are close to the sample features in the historical tasks. Compared with other existing incremental learning methods, the model trained by the present invention can overcome model forgetting and realize incremental learning without storing data and prototypes. The present invention proposes a task-oriented generation strategy that makes full use of the trained classifier, so that a lightweight feature generator can generate dynamic task features that meet incremental training. At the same time, the classification model of the present invention can achieve higher accuracy, making incremental learning technology for real-world applications possible.

[0075] According to one embodiment of the present invention, an image classification method is provided, wherein the method comprises: obtaining an image to be processed; and processing the image to be processed using an image classification model trained by any of the methods described in the above embodiments to obtain a classification result.

[0076] The image classification model based on incremental learning proposed in the present invention can be applied to a variety of real-world scenarios such as face recognition, traffic sign recognition, and medical image analysis. Taking the community face recognition access control system as an example, the model of the present invention can recognize the face information of new residents without loading any database, while ensuring the stable recognition of the face information of old residents. Compared with other existing methods, the method of the present invention can achieve good results, especially in the incremental learning image classification task with higher performance, and no need to store data or prototypes.

[0077] The various embodiments of this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments or implementations. The same or similar parts between the various embodiments of the present invention can be referred to each other, and the implementation principles of the inventive concept and the technical effects produced can be referred to each other, which will not be repeated here. In the absence of conflict, the various embodiments or implementations of the present invention can be combined with each other.

[0078] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0079] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0080] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0081] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used in the embodiments of the present invention is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed in the embodiments of the present invention.

Claims

1. An image classification model based on class incremental learning, comprising a solver and a feature generator; wherein: The solver includes a feature extractor, a classifier and a plurality of auxiliary classifiers; The feature extractor is used to extract features according to the input task and output the final features, and includes a plurality of learning modules for feature extraction connected in sequence, and the output of the previous learning module is the input of the next learning module; The classifier is used to classify the task according to the final feature; The auxiliary classifier is connected one-to-one with the outputs of other learning modules except the last learning module, and is used to generate embedding knowledge for embedding distillation according to the outputs of the learning modules during model training; as well as The feature generator is used to generate generated features of historical tasks under the guidance of the classifier, and fit the generated features with the current task features through loss to train the solver.

2. The model according to claim 1, characterized in that The feature extractor and classifier adopt the network structure of convolutional neural network ResNet-18; the feature generator includes a variational encoder.

3. A method for training an image classification model based on class incremental learning as claimed in claims 1-2, comprising an initial stage training and an incremental training stage, in, The initial stage training includes: Training the initial solver of the image classification model according to the original task obtained in the initial stage until convergence; Freeze the initial feature extractor and the initial classifier in the converged solver, and train the initial feature generator until convergence according to the original task in the initial stage; Each incremental training phase includes: Initialize the parameters of the image classification model trained in this stage using the parameters of the image classification model trained in the previous stage; The data of the original task input in this stage is extracted by the feature extractor obtained in the previous stage and then input into the feature generator trained in the previous stage to obtain the generated features of the original historical task; According to the generated features of the historical original task and the original task obtained in this stage, the solver in the image classification model initialized in this stage is trained until convergence; Freezing the initial feature extractor and the initial classifier in the converged solver; According to the solver trained for the original task of this stage and the feature generator trained in the previous stage, the feature generator in the initialized image classification model of this stage is trained until convergence.

4. The training method according to claim 3, characterized in that: In the initial stage training, the initial solver of the image classification model is trained until convergence according to the original task obtained in the initial stage, including: Train the initial feature extractor and the initial classifier according to the original task obtained in the initial stage, and calculate the first loss function; Perform enhancement processing according to the original task in the initial stage to obtain the enhanced task in the initial stage; Training the initial feature extractor and the initial auxiliary classifier according to the enhancement task in the initial stage, and calculating a second loss function; The sum of the first loss function and the second loss function is used as a third loss function, and based on the third loss function, the parameters of the initial feature extractor, the initial classifier, and the initial auxiliary classifier are updated until the initial solver converges.

5. The training method according to claim 3, characterized in that: In the initial stage training, freezing the initial feature extractor and the initial classifier in the converged solver, and training the initial feature generator according to the original task of the initial stage until convergence includes: Freeze the initial feature extractor and the initial classifier in the converged solver, input the original task of the initial stage into the initial feature generator, and obtain the generated features of the original task of the initial stage; Calculate a fourth loss function of the initial stage based on the final features output by the initial feature extractor, the generated features of the original task in the initial stage, the frozen parameters of the initial classifier, and the noise parameters; Based on the fourth loss function of the initial stage, the parameters of the initial feature generator are updated until the initial feature generator converges.

6. The training method according to claim 3, characterized in that: In each incremental stage of training, the parameters of the image classification model trained in the previous stage are used to initialize the parameters of the image classification model trained in the current stage, including: Initialize the solver trained in this stage with the parameters of the solver trained in the previous stage; The parameters of the feature generator trained in the previous stage are used to initialize the feature generator trained in this stage.

7. The training method according to claim 6, characterized in that: In each incremental stage of training, according to the generation features of the historical original task and the original task obtained in this stage, training the solver in the image classification model initialized in this stage until convergence includes: Input the original task obtained in this stage to the initialized solver and the solver trained in the previous stage respectively, and calculate the fifth loss function according to the output of the feature extractor in the initialized solver and the output of the feature extractor in the solver trained in the previous stage; Perform enhancement processing according to the original task obtained in this stage to obtain the enhanced task of this stage; Inputting the enhancement task of the present stage into the initialized solver and the solver trained in the previous stage respectively, and calculating the sixth loss function according to the output of the auxiliary classifier in the initialized solver and the output of the auxiliary classifier in the solver trained in the previous stage; Input the features extracted by the feature extractor in this stage into the feature generator trained in the previous stage to obtain the generated features of the historical original task; Based on the generated features of the historical original task and the original task of this stage, the initialized solver is trained to calculate the seventh loss function; A weighted sum of the fifth loss function, the sixth loss function and the seventh loss function is used as an eighth loss function, and based on the eighth loss function, parameters of the initialized solver are updated until the initialized solver converges.

8. The training method according to claim 7, characterized in that: In each incremental stage training, according to the solver trained for the original task of the current stage and the feature generator trained in the previous stage, training the feature generator in the image classification model initialized in the current stage to convergence includes: Initialize the feature generator of this stage according to the feature generator trained in the previous stage; Input the original task of the current stage into the initialized feature generator, and calculate the fourth loss function of the incremental stage; After inputting the original task of this stage into the solver trained by the original task of this stage, its output is then input into the initialized feature generator to calculate the ninth loss function; The weighted sum of the fourth loss function and the ninth loss function of the incremental stage is used as the tenth loss function, and the parameters of the initialized feature generator are updated based on the tenth loss function until the initialized feature generator converges.

9. The training method according to any one of claims 4 or 7, characterized in that: The original task is enhanced to obtain an enhanced task, including: The original task includes multiple images; The image is rotated and / or mirrored to obtain a processed image; The processed image constitutes an enhancement task.

10. A method for image classification, wherein: The method comprises: Get the image to be processed; The image to be processed is processed using an image classification model trained by the method as described in any one of claims 3 to 11 to obtain a classification result.

Images