An image classification method and system based on multi-normalization and dynamic network incremental learning

Abstract

A kind of image classification method based on class incremental learning of multi-normalization and dynamic network, comprising the following steps;Step 1: build new base network;Step 2: based on the initial image classification task D 1 of class incremental learning training of new base network, obtain initialization model, and freeze initialization model as the old model of next stage image classification task D 2 of class incremental learning;Step 3: when processing new class incremental learning image classification task, build feature enhancement model;Step 4: the size of feature enhancement model is controlled by model distillation technology, and the target compression model is obtained by distilling feature enhancement model;Target compression model is used as the old model when processing the next class incremental task;Step 5: on test sample, using target compression model carries out image classification precision detection.The present application is aimed at image classification task in the field of class incremental learning, used to relieve model catastrophic forgetting problem, improve the accuracy of model downstream image classification task.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer image classification, and particularly relates to an image classification method and system based on class-incremental learning of multi-normalization and dynamic network. BACKGROUND

[0002] Artificial neural networks acquire knowledge through generalized learning from different training stages and have achieved remarkable success in solving specific static classification tasks. However, with the change of training tasks in different real-world scenarios, expanding the static model built in the artificial neural network will cause the catastrophic forgetting problem, that is, the model will largely forget the knowledge of the old task after learning the new task. In this environment, lifelong learning emerges as the times require, which aims to change the mode of static network to a network that can continuously accumulate knowledge through different tasks without starting training from scratch each time. Class-incremental learning, as the most typical and most realistic scenario in lifelong learning, aims to train a model with limited memory size to meet the needs of the real world.

[0003] Currently, the most commonly used method in the field of class-incremental learning is usually based on knowledge distillation, such as the invention with the name "Class-incremental learning method based on multi-level knowledge distillation" and the application publication number (CN 117494790 A). Although the method based on knowledge distillation can align the outputs of the new model and the old model when training the new model, thereby preserving the knowledge of the old task and achieving the learning of the new task, there are still some limitations:

[0004] Firstly, since the distillation process relies on a fixed network, the use of a single backbone network often leads to insufficient model plasticity, lacking sufficient learning ability to cope with new classes;

[0005] Secondly, due to the limited access to old data, the features of old tasks may degenerate, leading to catastrophic forgetting and affecting the performance of old tasks.

[0006] In recent years, class-incremental learning methods based on dynamic networks have achieved good performance. These methods preserve the classification performance of old classes by freezing old modules and expand new modules based on the original model to improve the learning ability of new classes. However, with the continuous expansion of new task modules, the number of model parameters increases linearly, leading to increased training time, computational overhead, and memory consumption. If reasonable measures are not taken to optimize the preservation of old models, it may affect the performance of new models, thereby causing the overall model accuracy to decline. SUMMARY

[0007] In order to overcome the deficiencies of the prior art, the purpose of the present application is to provide a kind of image classification method and system based on multi-normalization and dynamic network class incremental learning, the present application is aimed at image classification task under the field of class incremental learning, by optimizing model structure and expansion mode, for relieving model catastrophic forgetting problem, improve the accuracy of downstream image classification task of model.

[0008] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0009] A kind of image classification method based on multi-normalization and dynamic network class incremental learning, comprising the following steps:

[0010] Step 1: introduce multi-normalization module on the basis of model backbone network, build new base network M base .

[0011] Step 2: based on the new base network, the training of initial image classification task D 1 Of class incremental learning, obtain initialization model, and freeze initialization model as old model of next stage image classification task D 2 Of class incremental learning;

[0012] Step 3: when processing new class incremental learning image classification task, based on the new base network, an additional new model is trained for new task, and the old model and the new model are connected through fully connected layer to build feature enhancement model;

[0013] Step 4: the size of the feature enhancement model is controlled by model distillation technology, and the target compression model is obtained by distilling the feature enhancement model;The target compression model is used as the old model when processing the next class incremental task;

[0014] Step 5: using the target compression model to detect the image classification precision on test sample.

[0015] Further, in step 1, the model backbone model adopts Resnet32 model, and the new multi-normalization module is constructed and the base network M base Specific steps are as follows:

[0016] Step 1.1: the class incremental learning scene is set to gradually learn in data stream with new class, assuming that there is a series of K image classification training tasks {D 1 ,D 2 ,…,D K}, There is no overlapping class between each task, wherein Indicates the t th Incremental stage, this stage contains n ta sample, while constructing a sample set ε with limited memory size to store the representative samples of the learned classes, and the feature map input to the multi-normalization module is denoted as where B is the mini-batch size, C is the number of channels, W is the width, and H is the height, which is divided into two parts along the channel dimension, to obtain the first feature map and the second feature map

[0017] Step 1.2: Apply the batch normalization (BN) method to the first feature map p1, calculate its mean and variance, and use affine transformation to obtain the normalized feature map That is,

[0018]

[0019] where, represents the mean, represents the variance, γ BN is the weighted balance factor;

[0020] Step 1.3: Apply the spatial normalization (SN) method, such as instance normalization (IN) or layer normalization (LN), to the second feature map p2, during which the corresponding mean and variance are calculated according to the selected spatial normalization type to obtain the feature map

[0021]

[0022] where SN = {IN, LN}, if SN selects IN, the mean along the channel dimension is calculated as and the variance is if SN selects LN, the mean along the layer dimension is calculated as and the variance is

[0023] When multiple spatial normalization methods are used, the balance factors w and w ′ are introduced, while in order to apply softmax later, it is required that k∈{SN} w k = 1 and k∈{SN} w′ k = 1, and β SN represents the affine transformation parameter of the spatial normalization method, and γ SN is the weighted balance factor;

[0024] Step 1.4: Concatenate the feature map obtained in step 1.2 and the feature map obtained in step 1.3 along the channel dimension, and provide the concatenated feature as input to the activation layer, thereby building a new base network Mbase ;

[0025] Further, the specific steps of step 2 of building an initialization model are as follows:

[0026] Step 2.1: using the base network M base built in step 1.4, training on the initial task D 1 , and obtaining an initialization model;

[0027] Step 2.2: freeze the initialization model in step 2.1 and use it as an old model M1 for training on the next incremental learning task D 2 .

[0028] Further, the specific steps of step 3 of expanding the model size to build a feature enhancement model are as follows:

[0029] Step 3.1: when a new image classification task D t ,t∈[2,K] arrives, first expand the old model M t-1 from the previous stage, add new model structure ΔM t to learn new categories, and adjust the network structure as follows:

[0030]

[0031] where, represents the expansion of the model, M t is the enhanced model after expansion, ΔM t is a model structure of the same size as M base , and M t has built a fully connected layer that can connect new and old models;

[0032] Step 3.2: after building the enhanced model M t , implement feature enhancement of the model in training using a reconstruction loss function, and the specific loss function is as follows:

[0033]

[0034] where, is the error between the prediction value of the new model M t and the actual value y, D KL (·||·) represents the difference between two probability distributions calculated by KL divergence, and S(·) represents the softmax operation; is the difference between the prediction value of the old model and the actual value, and γ is a hyperparameter for weighting; Use knowledge distillation to encourage the new model to have a similar output distribution as the old model on old categories;

[0035] Step 3.3: Utilize the constructed loss function On the dataset D t ∪ε, the model M t is class-incrementally trained.

[0036] Further, the step 4 compresses the model size, and the specific steps for building the target compressed distillation model are:

[0037] Step 4.1: For the feature enhancement model M t obtained in step 3, its size is twice that of M base , and here the compression means is adopted to reduce its size to the size of M base :

[0038] M t_new = Com(M t )

[0039] Where Com(·) represents model distillation, and the purpose is to remove redundant parameters and unnecessary dimensions in the enhanced model, and to maintain the size of the target compressed model consistent with M base ;

[0040] Step 4.2: For the Com(·) means in step 4.1, a balanced distillation algorithm is adopted, and the model loss formula here is:

[0041]

[0042] Where α is the weighting parameter, D KL (·||·) represents the KL divergence, and S(·) represents the softmax operation. Let the loss function join the training process to promote the target compressed model to fully learn the knowledge possessed by the enhanced model, thereby reducing the catastrophic forgetting;

[0043] Step 4.3: Use the loss mentioned in step 4.2 to train M t_new , and after training, M t_new is taken as the target compressed model M t of this stage;

[0044] Step 4.4: If the current task D t ∈{D 2 ,…,D K-1}, the target compressed model M t is used as the old model for the next stage of the task, thereby perfecting the entire class-incremental learning process; if the current task is the final task D K , the target compressed model is directly output as the result.

[0045] Further, the step 5 adopts image classification accuracy measurement to perform whole-process and phased performance evaluation on the class-incremental learning, and the specific steps of the classification accuracy prediction are as follows:

[0046] Step 5.1: Since the training environment of the model is class-incremental learning, the performance of the model trained for each task needs to be evaluated to determine the task performance and classification performance of the model in the whole incremental learning. As can be known from step 4.3, the target compression model M of the current stage can be obtained after the t-th task is trained. t ;

[0047] Step 5.2: For the t tasks that have been trained, the test sample set of all known classes is wherein l j represents the number of test samples of the j-th class, is the input sample, is the real label thereof, and the target compression model M t trained is used to predict the test samples of all tasks, and the output of the model is the predicted label For each task, the classification accuracy Acc is calculated by using the following formula j :

[0048]

[0049] wherein, is an indicator function, and takes the value of 1 when the predicted value is consistent with the real label , otherwise, 0.

[0050] Step 5.3: The average classification accuracy of the model on all tasks is calculated to evaluate the overall performance of the model in the incremental learning process:

[0051]

[0052] The value represents the comprehensive classification accuracy of the model on all tasks, which is a key indicator for measuring the catastrophic forgetting in the incremental learning.

[0053] According to a second aspect of the embodiments of the present application, an image classification system based on class-incremental learning of multi-normalization and dynamic network is provided, which comprises a data set unit, a basic network unit, a class-incremental learning unit and a classification performance evaluation unit.

[0054] For the image classification task under the class-incremental learning, the data set unit performs relevant processing on the image classification data set to obtain a training set, a validation set and a test set.

[0055] The basic network unit builds a multi-normalization module to optimize the backbone network, realizes the construction of the basic network, and simultaneously performs model initialization training using the training set and the validation set delivered by the data set unit;

[0056] The class-incremental learning unit performs class-incremental learning using the basic network built by the basic network unit and the training set and the validation set from the data set unit.

[0057] The final model obtained by the compression distillation module in the class-incremental learning unit is sent to the classification performance evaluation unit, and the performance of the model in the image classification task is evaluated using the test set.

[0058] The specific description of the units is as follows:

[0059] The data set unit is used to store and manage the data set of the class-incremental learning image classification task, including the training set, the validation set and the test set. This unit supports dynamic loading and segmentation of data, and can divide the data set into multiple incremental stages according to the task requirements, and provides data enhancement function to improve the generalization ability of the model.

[0060] The basic network unit builds a basic network containing a multi-normalization module, which dynamically selects a normalization method suitable for the characteristics of the task and completes the training of the basic network in the initial stage, providing strong feature learning ability for subsequent incremental tasks.

[0061] The class-incremental learning unit includes a dynamic network expansion module and a model distillation compression module, which is responsible for dynamic network expansion and compression when new tasks arrive, enhances the learning ability of new tasks through the addition of new modules, and at the same time preserves the classification performance of old tasks by freezing old modules. This unit combines knowledge distillation technology to optimize the balance between new and old tasks, effectively alleviating the problem of catastrophic forgetting.

[0062] The classification performance evaluation unit evaluates and monitors the classification performance of the model on all incremental tasks by calculating the classification accuracy of the test data set.

[0063] An image classification device based on class-incremental learning of multi-normalization and dynamic network, comprising:

[0064] Memory: for storing a computer program for implementing the image classification method based on class-incremental learning of multi-normalization and dynamic network;

[0065] Processor: for executing the computer program to implement the image classification method based on class-incremental learning of multi-normalization and dynamic network.

[0066] A computer readable storage medium, comprising:

[0067] The computer readable storage medium stores a computer program, and the computer program can realize an image classification method based on class-incremental learning of multi-normalization and dynamic network when executed by the processor.

[0068] The computer readable storage medium stores a computer program, and the computer program can realize an image classification method based on class-incremental learning of multi-normalization and dynamic network when executed by the processor.

[0069] The present application has the following beneficial effects:

[0070] The present application provides more flexible and effective feature processing capabilities for the model by designing a multi-normalization module, fully combining the advantages of batch normalization and spatial normalization, so that the model can dynamically adjust the feature distribution of new and old image classification tasks, significantly improving the learning performance and task adaptability of the model.

[0071] The present application improves the model's memory ability for old knowledge by retaining the old image classification task model, and adds a dedicated module for the new image classification task to enhance the model's learning ability for the new task, while ensuring the performance of the new task and improving the model's performance on the old task, achieving balanced learning of new and old image classification tasks.

[0072] The present application designs a new loss function, dynamically adjusts the learning direction of the model from the perspectives of new task learning, old knowledge retention and comparison between new and old models, optimizes the balance between new task learning and old task memory, and ensures that the overall performance of the model in the multi-task scenario does not fluctuate significantly.

[0073] The present application distills the feature enhancement model to reduce redundant parameters and additional overhead, controls the size of the model, and ensures the memory of the model on the whole task, making the model more practical.

[0074] The present application significantly enhances the stability and accuracy of the model in processing image classification tasks through the collaborative application of multiple technologies, and realizes the comprehensive improvement of the learning efficiency and image classification accuracy of the model.

[0075] The present application combines dynamic network and knowledge distillation, model compression and precision prediction technology, and the overall system has low computational overhead and memory requirement, is suitable for class-incremental learning scenarios, and can effectively alleviate the catastrophic forgetting problem.

[0076] The dynamic network strategy of the application is that the model size is expanded or shrunk during the training process. In step 3, when a new task of class incremental learning comes, an additional network architecture is trained for the new task, so as to realize the expansion of the model size, that is, the old model + the new model, and the model size is expanded to twice the original model. In step 4, after the model is expanded, in order to keep the model size from continuously increasing (because if a model is trained for each task, the number of models and the size of parameters will increase linearly, causing excessive consumption of computing resources and memory resources), the knowledge distillation method is used to distill the twice model size into one, realizing the shrinkage of the model size. Through the expansion and shrinkage, the dynamic network technology is realized. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 The flowchart provided for the application.

[0078] Figure 2 The multi-normalization module and the base network schematic diagram provided for the application.

[0079] Figure 3 The learning step decomposition diagram provided for the application.

[0080] Figure 4 The B0-5 task incremental learning accuracy comparison line graph of the application and other methods on the CIFAR100 dataset.

[0081] Figure 5 The B0-10 task incremental learning accuracy comparison line graph of the application and other methods on the CIFAR100 dataset.

[0082] Figure 6 The system schematic diagram provided for the embodiment of the application. DETAILED DESCRIPTION

[0083] The application will be further described in detail below with reference to the accompanying drawings.

[0084] Reference Figure 1 The image classification method flowchart of class incremental learning based on multi-normalization and dynamic network provided for the application can alleviate the catastrophic forgetting problem in the class incremental learning scene, effectively control the model size, and realize better model performance. By comprehensively utilizing the advantages of multiple normalization methods to design a multi-normalization module, the network is optimized from the module composition level of the model, and the catastrophic forgetting problem is alleviated. In addition, based on the dynamic network, a two-stage training model of feature enhancement-compression distillation is built, which retains the advantages brought by model expansion on the one hand, and effectively avoids the negative effects brought by parameter size expansion and old model reservation on the other hand. The method consists of the following steps:

[0085] In the process of invention verification, the image data set CIFAR100 containing 100 categories is used, and experiments are carried out based on the traditional backbone network Resnet32. The initial model and the feature enhancement model are trained for 150 iterations, and the compressed model is trained for 130 iterations. In the training process, the batch size is set to 128, the random gradient descent optimizer is used, the initial learning rate is 0.1, and the cosine annealing is used to adjust the learning rate in the training process. There are two settings in the experiment, B0 and B50, which means that the initial stage model does not learn any category and the initial stage learns 50 categories. The B0-10 task under the CIFAR100 data set means that no category is learned in the initial stage, and 10 categories are learned in each of the following 10 task stages.

[0086] Step 1: Introduce a multi-normalization module to replace the traditional single batch normalization method based on the model backbone network, and build a new basic network M base ;

[0087] Step 1.1: The class incremental learning scenario is set to gradually learn in the data stream with new categories. A series of K training tasks {D 1 ,D 2 ,…,D K} are set, and there is no overlapping class between each task. Where represents the t-th incremental stage, which contains n t samples, and a memory with limited size ε is constructed to store the representative samples of the learned categories.

[0088] Referring to Figure 2 , the multi-normalization module and the basic network diagram, the left side is the specific composition of the multi-normalization module, and the right side is the specific position of the multi-normalization module in the basic network M base ;

[0089] The multi-normalization module receives the feature map from the convolutional layer, where the feature map input to the multi-normalization module is denoted as , where B is the batch size, C is the channel number, W is the width, and H is the height. It is divided into two parts along the channel dimension, respectively and

[0090] Step 1.2: Apply the batch normalization (BN) method to the first feature map p1, calculate its mean and variance, and use affine transformation to get the normalized feature map , that is:

[0091]

[0092] where, denotes the mean, denotes the variance, γ BN is the weighted balance factor;

[0093] Step 1.3: Apply a spatial normalization (SN) method, such as instance normalization (IN) or layer normalization (LN), to the second feature map p2, in the process, according to the selected spatial normalization type, the corresponding mean and variance are calculated to obtain the feature map

[0094]

[0095] where, SN = {IN, LN}, if SN selects IN, there is a mean along the channel dimension and variance if SN selects LN, there is a mean along the layer dimension and variance When multiple spatial normalization methods are used, the balance factors w and w' are introduced ′ , and in order to apply softmax later, it is required that k∈{SN} w k = 1 and k∈{SN} w' k = 1, and β SN denotes the affine transformation parameter of the spatial normalization method, γ SN is the weighted balance factor;

[0096] Step 1.4: Concatenate the feature map obtained in step 1.2 and the feature map obtained in step 1.3 along the channel dimension, as input to the activation layer, thereby building a new base network M base ;

[0097] Referring to Figure 3 , the method provided by the present application provides step one of the learning step decomposition diagram. Through step one, the training of the base network M base on the task D 1 is realized, thereby obtaining an initialized model.

[0098] Step 2: In the first phase of class-incremental training, based on the network M base , the initial task D 1 is trained, for example, the D 1 of the B0-10 task under the CIFAR100 data set represents the learning of the first 10 classes, and an initialized model is obtained;

[0099] Step 2.1: Use the Mbase , perform initial task D 1 on training, and get the initialization model;

[0100] Step 2.2: freeze the initialization model in step 2.1, and use it as an old model M1 for the next class incremental learning task D 2 training;

[0101] Referring to Figure 3 , the learning step of the method provided by the application is divided into step two of the diagram. Through step two, a new model structure can be trained for a new image classification task, while a feature enhancement model is built together with the frozen model of the previous stage, so as to improve the accuracy of the model for image classification tasks.

[0102] Step 3: when processing a new class incremental task, based on the base network, a new model is additionally trained for the new task, a full connection layer is constructed, and the old model and the new model are effectively connected through the full connection layer to build a feature enhancement model;

[0103] Step 3.1: when a new task D t ,t∈[2,K] comes, first, the model M t-1 of the previous stage is expanded, and a new model structure ΔM t is added to learn new classes, and the adjustment of the network structure is represented as:

[0104]

[0105] wherein, represents the expansion of the model, M t is the enhanced model after expansion, ΔM t is a model structure with the same size as M base , and M t has a full connection layer that can connect the new and old models;

[0106] Step 3.2: after the construction of the enhanced model M t , the feature enhancement of the model is realized by using the reconstruction loss function in the training, and the specific loss function is:

[0107]

[0108] wherein, is the error between the prediction value of the new model M t and the actual value y of the data, D KL (·||·) represents the difference between two probability distributions calculated by KL divergence, and S(·) represents the softmax operation; is the difference between the prediction value of the old model and the actual value, and γ is a hyperparameter for weighting; Knowledge distillation is utilized to encourage the new model to have a similar output distribution as the old model on the old classes;

[0109] Step 3.3: Utilize the constructed loss function On the dataset D t ∪ε, the model M t is class-incrementally trained;

[0110] Referring to Figure 3 , the present application provides step three of the learning step breakdown of the method. Through step three, the feature enhancement model size can be reduced to obtain a target compression model by using model distillation technology, so as to effectively control the model size while not losing the image classification accuracy.

[0111] Step 4: Control the size of the feature enhancement model by using model distillation technology, and distill the feature enhancement model to obtain a target compression model; the target compression model serves as an old model when processing the next class-incremental task;

[0112] Step 4.1: For the feature enhancement model M t obtained in step 3, its size is twice that of M base , and here the compression means is used to reduce its size to be equal to that of M base :

[0113] M t_new =Com(M t )

[0114] Wherein, Com(·) represents model distillation, and the purpose is to remove redundant parameters and unnecessary dimensions in the enhancement model, so that the size of the target compression model is consistent with M base ;

[0115] Step 4.2: For the Com(·) means in step 4.1, a balanced distillation algorithm is used, and the model loss formula here is:

[0116]

[0117] Wherein, α is a weighting parameter, D KL (·||·) represents KL divergence, and S(·) represents a softmax operation. The loss function is added to the training process to promote the target compression model to fully learn the knowledge possessed by the enhancement model, thereby reducing the catastrophic forgetting;

[0118] Step 4.3: Use the loss mentioned in step 3.2 to train M t_new , and after the training is completed, M t_new is taken as the target compression model M t of this stage;

[0119] Step 4.4: If the current task D t ∈{D 2 ,…,D K-1}, the target compressed model M t is used as the old model for the next stage task, so as to complete the whole class-incremental learning process; if the current task is the final task D K , the target compressed model is directly output as the result.

[0120] Step 5: On the test samples, the image classification accuracy of the target compressed model is detected.

[0121] Step 5.1: Since the training environment of the model is class-incremental learning, it is necessary to evaluate the performance of the stage model trained for each task, so as to determine the task performance and classification performance of the model in the whole incremental learning. As can be seen from step 4.3, after the training of the t-th task is completed, the target compressed model M t of the stage can be obtained.

[0122] Step 5.2: For the t tasks that have been trained, the test sample set of all known classes is wherein l j denotes the number of test samples of the j-th class, is the input sample, is the corresponding real label, and the test samples of all tasks are predicted using the trained target compressed model M t , and the output of the model is the predicted label For each task, the classification accuracy Acc j is calculated using the following formula:

[0123]

[0124] wherein, is an indicator function, which takes the value of 1 when the predicted value is consistent with the real label , and 0 otherwise.

[0125] Step 5.3: The average classification accuracy of the model on all current tasks is calculated to evaluate the overall performance of the model in the incremental learning process:

[0126]

[0127] This value represents the comprehensive classification accuracy of the model on all tasks, which is a key indicator for measuring catastrophic forgetting in incremental learning.

[0128] To evaluate the performance of the proposed method, we compare our method with several state-of-the-art methods, including iCaRL, BiC, WA, COIL, PODNet, DER, AFC, MEMO, and eTag. We conduct experiments on CIFAR100 dataset with 100 classes, and test the performance of the models in different incremental stages. Specifically, we test the performance of the models in five settings, including B0 and B50:

[0129] Setting 1: B0-5 tasks. B0 represents that the model has not learned any classes in the initial stage, i.e., the initial task contains 0 classes. Then, the model learns 5 image classification tasks in the incremental stage, each containing 20 classes.

[0130] Setting 2: B0-10 tasks. B0 also represents that the initial task contains 0 classes in the initial stage. Then, the incremental stage divides the dataset into 10 tasks, each containing 10 classes.

[0131] Setting 3: B0-20 tasks. B0 also represents that the initial task contains 0 classes in the initial stage. Then, the incremental stage divides the dataset into 20 tasks, each containing 5 classes.

[0132] Setting 4: B50-5 tasks. B50 represents that the model has learned 50 classes in the initial task. Then, the incremental stage further divides into 5 tasks, each learning 10 classes of the remaining 50 classes.

[0133] Setting 5: B50-10 tasks. B50 represents that the model has learned 50 classes in the initial task. Then, the incremental stage divides the remaining 50 classes into 10 tasks, each learning 5 classes.

[0134] The experimental results are reported in Table 1, which shows the average accuracy of the models in all stages.

[0135] Table 1: Comparison of average accuracy of multiple methods on CIFAR100

[0136]

[0137]

[0138] Referring to Table 1, the method (Ours) proposed in the present application performs well in settings 2-5 of the CIFAR100 dataset, and has performance improvement compared with various advanced strategies, fully proving the effectiveness and advantages of the method in the class-incremental learning task. Specifically, the method introduces a multi-normalization module, a dynamic network expansion and compression strategy, and an optimized loss function design, effectively alleviates the catastrophic forgetting problem while taking into account the learning ability of new tasks, and realizes a good balance between new and old tasks.

[0139] Referring to Figure 4 , the accuracy comparison line chart of the method of the present application and other methods when performing B0-5 task incremental learning on the CIFAR100 dataset. The figure shows the recording and drawing results of the image classification accuracy for the whole stage under setting 1. As can be seen from the figure, the overall accuracy of the method proposed in the present application is better than that of other comparison methods in the incremental learning of each stage, which shows the advantages of the present application.

[0140] Referring to Figure 5 , the accuracy comparison line chart of the method of the present application and other methods when performing B0-10 task incremental learning on the CIFAR100 dataset. Through comparison under experimental setting 2, it can be further verified that the method of the present application still maintains relatively stable and high accuracy in a larger task range, and continues to lead other existing methods.

[0141] Referring to Figure 6 , a class-incremental learning system based on multi-normalization and dynamic network provided by an embodiment of the present application is shown in the schematic diagram, which includes the following modules:

[0142] Dataset unit: used for storing and managing the dataset of class-incremental learning task, containing training set, validation set and test set. This unit supports dynamic loading and segmentation of data, and can divide the dataset into multiple incremental stages according to the task requirements, and provides data enhancement function to improve the generalization ability of the model.

[0143] Basic network unit: a basic network containing a multi-normalization module is constructed, and this unit dynamically selects the normalization method suitable for the task characteristics and completes the training of the basic network in the initial stage, providing strong feature learning ability for subsequent incremental tasks.

[0144] Class-incremental learning unit: containing a dynamic network expansion module and a model distillation compression module, i.e. responsible for dynamic network expansion and compression when a new task comes, enhancing the learning ability of new tasks through the addition of new modules, while preserving the classification performance of old tasks by freezing old modules. This unit optimizes the balance between new and old tasks by combining knowledge distillation technology, effectively alleviating the catastrophic forgetting problem.

[0145] Classification performance evaluation unit: the classification performance of the model on all incremental tasks is evaluated and monitored by calculating the classification accuracy of the test data set.

Claims

1. A method for image classification based on multi-normalization and dynamic network-based class-incremental learning, characterized in that, Comprising the following steps: Step 1: Introduce multi-normalization module on the basis of the model backbone network, build a new base network M base ; Step 2: Class-incremental learning initial image classification task D based on the new base network 1 training, get the initialization model, and freeze the initialization model as the old model of the next stage of image classification task D 2 class-incremental learning Step 3: When processing a new class-incremental learning image classification task, an additional model is trained for the new task based on the new base network, and a feature enhancement model is built by connecting the old model and the new model through a fully connected layer; Step 4: The size of the feature enhancement model is controlled by model distillation technology, and the target compression model is obtained by distilling the feature enhancement model; the target compression model serves as the old model when processing the next class-incremental task; Step 5: On the test sample, the target compression model is used for image classification accuracy detection; The step 1, the model backbone model adopts the Resnet32 model, constructs a new multi-normalization module, and builds the basic network M on the basis base The specific steps are as follows: Step 1.1: The class-incremental learning scenario is set to learn step by step in the data stream with new classes, and there are a series of K image classification training tasks {D 1 ,D 2 ,…,D K} without overlapping classes between each task, where represents the t-th incremental stage, which contains n t samples, and a limited memory size sample set ε is constructed to store representative samples of learned classes, and the feature map input to the multi-normalization module is denoted as where B is the batch size, C is the channel number, W is the width, and H is the height, which is divided into two parts along the channel dimension, respectively, to obtain the first feature map and the second feature map Step 1.2: Apply the batch normalization (BN) method to the first feature map p1, calculate its mean and variance, and use the affine transformation to obtain the normalized feature map That is: wherein, denotes the mean, denotes the variance, γ BN is a weighted balancing factor; Step 1.3: Apply spatial normalization (SN) method to the second feature map p2, and calculate the corresponding mean and variance according to the selected spatial normalization type to obtain the feature map where SN = {IN, LN}, if SN chooses IN, the mean along the channel dimension is calculated and variance if SN chooses LN, the mean along the layer dimension is calculated and variance In the case of multiple spatial normalization methods, balance factors w and w' are introduced ′ with ∑ k∈{SN} w k = 1 and ∑ k∈{SN} w' k = 1, while β SN represents the affine transformation parameter of the spatial normalization method, and γ SN is the weighted balance factor. Step 1.4: Concatenate the feature maps obtained in step 1.2 and step 1.3 along the channel dimension, and provide the concatenated features as input to an activation layer to build a new base network M along the channel dimension, and provide the concatenated features as input to an activation layer to build a new base network M base . 2.The image classification method based on multi-normalization and dynamic network incremental learning according to claim 1, wherein, The specific steps of building the initialization model of step 2 are: Step 2.1: Use the base network M built in Step 1.4 to perform training on the initial task D and obtain an initialized model. base Step 2.2: Fine-tune the initialized model on the target task T. 1 Step 2.3: Obtain the fine-tuned model. Step 2.2: Freeze the initialized model in Step 2.1 and use it as the old model M1 for the next class of incremental learning tasks D 2 training. 3.The image classification method based on multi-normalization and dynamic network incremental learning according to claim 2, wherein, The specific steps of expanding the model size of step 3 to build the feature enhancement model are: Step 3.1: When a new image classification task D t arrives, the old model M t-1 is first extended to add new model structure AM t for learning the new classes, the adjustment of network structure is represented as: wherein, represents an extension of the model, M t is the extended enhanced model, AM t is the model M base is a model structure of the same scale, M t a fully connected layer is built on top that can relate the new and old models; Step 3.2: In the enhanced model M t After the completion of the construction, the feature enhancement of the model is realized in the training by using the reconstruction loss function. The specific loss function is: wherein, is the error between the prediction of the new model M t and the actual value of the data y, D KL (·||·) denotes the difference between two probability distributions calculated by KL divergence, S(·) denotes the softmax operation; is the difference between the prediction of the old model and the actual value, and γ is a hyperparameter for weighting; Knowledge distillation is used to encourage the new model to have a similar output distribution as the old model on the old classes. Step 3.3: Utilizing the constructed loss function On the dataset D t ∪ε pair model M t is class-incrementally trained.

4. The image classification method based on multi-normalization and dynamic network incremental learning according to claim 3, characterized in that, The specific steps of compressing the model size of step 4 to build the target compression distillation model are: Step 4.1: For the feature enhancement model M t reducing its size to be of the same order as M base of the same order: M t_new = Com(M t ) where Com(·) denotes model distillation, removing redundant parameters and unnecessary dimensions in the enhanced model, maintaining the target compression model size with M base Consistency is maintained; Step 4.2: For the Com(·) method in step 4.1, a balanced distillation algorithm is used, and the model loss formula is: where a is a weighting parameter, D KL (·||·) denotes the KL divergence, S(·) denotes the softmax operation; Step 4.3: Utilize the loss mentioned in step 4.2 M t_new is trained, and after the training is completed, M t_new is taken as the target compression model M t of this stage; Step 4.4: If the current task D t ∈{D 2 ,…,D K-1}, the target compressed model M t is used as the old model for the next stage task, so as to complete the whole class-incremental learning process; if the current task is the final task D K , the target compressed model is directly output as the result.

5. The image classification method based on multi-normalization and dynamic network incremental learning according to claim 1, characterized in that, Step 5 uses image classification accuracy measurement to evaluate the performance of class-incremental learning in the whole process and in stages, and the specific steps of classification accuracy prediction are: Step 5.1: The target compression model M of this stage is obtained after the t-th task training is completed according to step 4.3 t ; Step 5.2: For the t tasks that have been trained, the test sample set of all known classes is where l j denotes the number of test samples of the jthclass, is the input sample, is its corresponding true label, and the trained target compression model M t is used to predict the label For each task, the classification accuracy Acc is calculated using the following formula j : wherein, is an indicator function that takes the value 1 when the predicted value is consistent with the true label and 0 otherwise; Step 5.3: Calculate the average classification accuracy of the model on all tasks, which is used to evaluate the overall performance of the model in the incremental learning process: This value represents the comprehensive classification accuracy of the model on all tasks, which is a key indicator for measuring catastrophic forgetting in incremental learning.

6. A multi-normalization and dynamic network based incremental learning like image classification system for implementing the method of any one of claims 1-5, characterized in that, It includes a data set unit, a base network unit, a class-incremental learning unit, and a classification performance evaluation unit; For image classification tasks under class-incremental learning, the data set unit processes the image classification data set to obtain the training set, validation set and test set; The base network unit builds a multi-normalization module to optimize the backbone network, realizes the building of the base network, and performs model initialization training using the training set and validation set delivered by the data set unit; The class-incremental learning unit uses the base network built by the base network unit and the training set and validation set from the data set unit for class-incremental learning; The final model obtained by the compression distillation module in the class-incremental learning unit is sent to the classification performance evaluation unit, and the performance of the model in the image classification task is evaluated using the test set.

7. The image classification system based on multi-normalization and dynamic network incremental learning according to claim 6, wherein, The data set unit is used to store and manage the data set of the class-incremental learning image classification task, and the data set is divided into multiple incremental stages according to the task requirements; The base network unit dynamically selects the normalization method suitable for the task characteristics and completes the training of the base network in the initial stage; The class-incremental learning unit includes a dynamic network expansion module and a model distillation compression module, which are responsible for dynamic network expansion and compression when a new task arrives; The classification performance evaluation unit evaluates and monitors the classification performance of the model on all incremental tasks by calculating the classification accuracy of the test data set.

8. An image classification device based on multi-normalization and dynamic network-based incremental learning, characterized by, It includes: Memory: for storing the computer program implementing the image classification method based on multi-normalization and dynamic network class-incremental learning of any one of claims 1-6; Processor: configured to implement the method of claim 1-6 when the computer program is executed.

9. A computer readable storage medium comprising: The computer readable storage medium stores a computer program, and the computer program is configured to implement the method of claim 1-6 when executed by a processor.

Images