Model knowledge-introduced synthetic playback linear increment continuous learning method

Abstract

The invention relates to a model knowledge-introduced synthetic playback linear increment continuous learning method, which comprises the following steps of: receiving a new task sample, downsampling and storing the new task sample, and initializing a synthetic sample; using data set distillation to update a synthetic sample, loss being gradient matching and relation matching loss, and updating synthetic model parameters at the same time; training the classification model in two stages: learning a new task sample and a sampled cache region sample in the first stage, and learning all samples in the cache region in the second stage; in the two stages, a soft label distillation mode is used for learning knowledge of a synthesis model so as to improve the plasticity of a classification model; and after the classification model completes incremental task learning, the average classification precision is tested. The knowledge of the synthetic model is introduced through the knowledge distillation technology, the plasticity of the classification model can be effectively enhanced, the model stability and plasticity are reasonably balanced, and the average classification precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical fields of image classification and continual learning in the field of image processing, and particularly relates to a synthetic replay online class-incremental continual learning method introducing model knowledge. Background Art

[0002] In recent years, deep neural networks have achieved great success in the field of image classification and even outperformed humans in some scenarios. However, deep neural networks usually assume that all data are independently and identically distributed, and the task boundaries are known and remain unchanged at the beginning, which does not hold in most real-world scenarios, such as gradually learning new classes under the condition of inaccessible previous datasets, data distribution changing over time, etc. In these scenarios, the model will quickly overwrite the learned knowledge after learning new knowledge, resulting in catastrophic forgetting. To solve this problem, continual learning methods have been proposed. Continual learning sequentially learns a series of tasks, aiming to maintain good performance on old tasks after the model has completed learning new tasks. Online class-incremental learning is a challenging setting in the field of continual learning, where the model needs to continuously learn new classes from the task data stream and each sample can only be accessed once.

[0003] Existing online class-incremental continual learning mainly includes methods based on regularization, methods based on network structure, and methods based on memory replay, etc.

[0004] Regularization methods penalize excessive changes of the model on new tasks by adding regularization terms to the loss function, thereby retaining the knowledge of old tasks. EWC measures the importance of parameters through the Fisher information matrix, and stronger regularization will be imposed on parameters that are more important for old tasks. MAS accumulates the changes of each parameter in the entire update history to measure the importance of the parameter. Although parameter regularization methods can prevent catastrophic forgetting without using old task data, in the context of long-sequence tasks, simply adding a parameter regularization term during training is not sufficient to confine the optimization process within the feasible region of old tasks. LwF introduces knowledge distillation into continual learning and uses the soft labels generated by the old model on new task samples to guide the learning process of the model on new tasks, providing a new idea for subsequent research. On this basis, LwM proposes to use the attention of the old model to guide the new model. To eliminate the prediction bias caused by only using new task samples during the knowledge distillation process, DMC proposes a method of integrating old and new models using unlabeled data. To better achieve the trade-off between stability and plasticity, ANCL uses an additional auxiliary model to promote the plasticity of the method mainly focused on enhancing stability.

[0005] Network structure-based methods can be divided into parameter isolation and dynamic structure methods. The parameter isolation method is characterized by allocating an isolated parameter subspace for each task, thus avoiding direct influence between different tasks. WSN explicitly optimizes a binary mask to select dedicated neurons or parameters for each task and freezes the masked areas of old tasks to prevent catastrophic forgetting. NISPA explicitly identifies important neurons or parameters of the current task and freezes them, and then allows the model to optimize the unimportant parts in subsequent tasks. However, these methods usually require sparsity constraints on model parameters and selective reuse of frozen parameters, which may affect the learning of each task. The dynamic structure method allows the model scale to expand by dynamically changing the network structure to allocate different parameters for different tasks. CCGN uses a gating network to add an additional convolutional layer for each new task, and this convolutional layer is only optimized for new task samples. DER proposes a two-stage learning method. In the first stage, a super feature extractor is trained by training a feature extractor for each task. In the second stage, the super feature extractor is frozen and the model is allowed to adjust the classifier on a class-balanced training subset. However, as the number of tasks continues to increase, the dynamic structure method will continuously add new components or branches, resulting in a gradual increase in memory occupancy and computational complexity, affecting its application in some scenarios.

[0006] Memory replay methods usually construct a buffer area to store previous task samples. When a new task arrives, the samples in the buffer will be used together with the new task samples for model update, enabling the model to review the knowledge of old tasks while learning new tasks and alleviating catastrophic forgetting. In practical applications, storage resources are limited. To avoid the linear expansion of the buffer area, the size of the buffer is fixed. Therefore, the number of samples of each class in the buffer will decrease as the number of tasks increases, which poses higher requirements for the effectiveness of the buffered samples. To improve the effectiveness of the buffered samples, early methods would select representative samples for storage and replay. Herding selects the samples closest to the class mean to store in the buffer. RWalk suggests selecting samples with high predictive entropy and close to the decision boundary. By replaying these difficult samples, the model will obtain higher generalization ability. Rainbow Memory proposes to evaluate the difficulty of samples through data augmentation and compare the predictions of multiple augmented samples to select samples with large prediction differences for replay. Similarly, PRN proves that selecting the samples with the largest diversity of gradient directions for replay can alleviate forgetting on the premise that the loss of memory samples does not increase during the current update. However, these methods all select replay samples based on the observation of the current model. The limited memory samples may not always meet the continuously updated selection requirements, resulting in low-quality sample replay. Some subsequent work proposed parameterizing memory samples and updating the memory samples to improve their representativeness. Mnemonics proposed a method to optimize memory samples in a meta-learning manner. This framework is trained through a two-layer optimization at the model level and example level and can be combined with various continuous learning algorithms. SSD uses the dataset distillation method to synthesize memory samples in order to make the limited memory samples contain more information. However, although these methods improve the stability of the model, the utilization efficiency of the buffer area is not high. They do not pay attention to the model knowledge of the synthesized model in dataset distillation. The plasticity of the online continuous learning model is insufficient, and it cannot effectively adjust plasticity and stability. At the same time, due to the small number of synthesized samples and multiple learning, it is easy to cause overfitting problems. Summary of the Invention

[0007] Technical Problems to be Solved

[0008] To avoid the deficiencies of the prior art, the present invention proposes a synthetic replay online class-incremental continuous learning method introducing model knowledge to improve the utilization efficiency of the buffer area, enhance the plasticity of the model under continuous learning tasks, effectively adjust the plasticity and stability of the model, alleviate the overfitting problem of synthetic samples, and improve the final average classification accuracy of online class-incremental continuous learning.

[0009] Technical Solution

[0010] A synthetic replay online class incremental continuous learning method for introducing model knowledge, characterized in that: the total number of tasks for continuous learning is N, and the task sequence is: T1,..., T i ,..., T N , and the incremental continuous learning steps are as follows:

[0011] Construct a buffer with a memory size of v. For each batch of newly received tasks T i The samples are downsampled by s times to obtain real samples. For each class, s*v / n real sample images are initialized to obtain synthetic samples, where s is the image downsampling factor and n is the total number of classes of all incremental tasks; the real samples and synthetic samples are stored in a buffer;

[0012] Use the dataset distillation method of gradient matching to update the synthetic samples to obtain the distilled synthetic samples;

[0013] Randomly sample the real samples and the distilled synthetic samples in the buffer, upsample the sampled samples by s times to obtain the sampled samples that restore the original image size, and combine the samples with the new tasks T i The samples are combined to obtain enhanced and augmented data samples; the enhanced and augmented data samples are input into the classification model for training of the classification model to complete the incremental task training of this batch;

[0014] For the classification models of all batches of newly trained tasks T i Test the classification models;

[0015] After all incremental tasks are trained, the classification accuracies of all tasks are aggregated and averaged to obtain the average classification accuracy of all tasks.

[0016] The downsampling by s times uses bicubic interpolation. The formula for calculating the pixel value of the interpolation point by bicubic interpolation is:

[0017]

[0018] where x and y are the coordinate values of the interpolation point, normalized to the range [0, 1], P(x, y) represents the pixel value of the interpolation point, and a ij is a coefficient calculated from the pixel values and derivatives of 16 known points around the interpolation point, x i y j is a polynomial basis function representing the product of the i-th power of x and the j-th power of y.

[0019] When the buffer receives a new task sample batch, if the memory is full, the real samples in it will be randomly replaced by the new task samples, while the synthetic samples are updated after initialization and are not replaced.

[0020] The dataset distillation method using gradient matching is as follows:

[0021] The synthetic samples are updated by minimizing the distance between the gradients of the synthetic model on the synthetic samples and the real samples. On the basis of the original loss, a relationship matching loss is added to make the synthetic samples and real samples of the new task categories have a consistent relationship with the synthetic samples of the old tasks; during the dataset distillation process, the update of the synthetic model parameters and the synthetic samples alternates. Whenever a batch of new task samples arrives, the synthetic model will perform a parameter update. Every κ times the synthetic model parameters are updated, the synthetic samples are updated once; when the synthetic model parameters are updated, only the real samples and new task samples in the buffer are received, and the synthetic samples in the buffer are not received to avoid strong coupling between the synthetic model parameters and the synthetic samples.

[0022] During dataset distillation, before distilling each task dataset, the output dimension of the classification head of the synthetic model becomes the number of classes of all classes seen before plus the number of classes of this task, and the synthetic model parameters are re-initialized.

[0023] The formula for the relationship matching loss is:

[0024]

[0025] where represents the average feature of the synthetic samples of class c obtained by the synthetic model, represents the average feature of the downsampled real samples of class c obtained by the synthetic model, represents the average feature of all synthetic samples of the old tasks obtained by the synthetic model, and dist represents the distance metric, taking the L2 norm.

[0026] The final dataset distillation loss is:

[0027] L dd = L g + αL r

[0028] where L g is the gradient matching loss, and α is the weight of the relationship matching loss.

[0029] The training of the classification model is divided into two stages; in the first stage, to retain the knowledge of the old tasks, some samples are sampled from the buffer, upsampled by s times, restored to the original image size, and then the buffer samples are merged with the new task sample batches, and then data augmentation is used for expansion to obtain the training input; the training loss of the classification model includes cross-entropy loss, contrastive learning loss, and knowledge distillation loss; among them, the knowledge distillation loss introduces the knowledge of the synthesis model through the method of soft-label knowledge distillation, enhances the ability to regulate the plasticity of the classification model, and since the synthesis model has not been updated on the synthetic samples, the knowledge of the synthesis model can also help alleviate the overfitting of the classification model to the synthetic samples; after the new task samples are learned, the second stage of training is entered, and the classification model learns all the samples in the buffer, and through learning the buffer samples, the classification model further learns the new task and old task knowledge, improving the classification accuracy.

[0030] During the training of the classification model, if the number of samples in the buffer is insufficient, the classification model training is not performed.

[0031] The training loss of the classification model is:

[0032] L = L contrastive + L ce + λL kd

[0033] where L contrastive is the contrastive learning loss, L ce is the cross-entropy loss, L kd is the knowledge distillation loss, and λ represents the coefficient of the distillation term;

[0034] The formula for the contrastive learning loss is:

[0035]

[0036] where B represents the training input of the classification model, z i represents the projection of sample i in B generated by the classification model, P(i) represents the set of positive samples of sample i in B, that is, the samples with the same class as sample i, A(i) represents the set of samples in B except sample i, and τ represents the temperature coefficient;

[0037] The cross-entropy loss is:

[0038]

[0039] where y i represents the one-hot encoding of sample i, and q i represents the predicted label of sample i;

[0040] The formula for the knowledge distillation loss is:

[0041]

[0042] Among them, B represents the training input of the classification model, and q i represents the predicted label of the classification model for sample i, ο i represents the predicted label of the synthesis model for sample i.

[0043] When the classification model is tested, the samples in the buffer are upsampled by s times first, then the feature extractor of the classification model extracts the features of all samples in the buffer, then the mean features of all classes are obtained by class, and finally the features of the test samples are calculated. The test samples are classified into the class with the mean feature closest to their features; the performance of the continuous learning model is usually evaluated by the average precision. After the classification accuracy of each task is statistically calculated, the average precision of the classification model is calculated; the test samples include all classes of the incremental tasks seen by the model.

[0044] Beneficial effects

[0045] A synthesis replay online class-incremental continuous learning method introducing model knowledge proposed by the present invention includes four steps: 1. Construct a dynamic buffer, receive new task samples, downsample and store them, and then initialize the synthesis samples; 2. Use dataset distillation to update the synthesis samples, with the loss being the gradient matching and relationship matching losses, and at the same time update the synthesis model parameters; 3. Train the classification model in two stages: the first stage learns the new task samples and the sampled buffer samples, and the second stage learns all the samples in the buffer. In both stages, the knowledge of the synthesis model is learned in the way of soft label distillation to improve the plasticity of the classification model; 4. After the classification model completes the incremental task learning, test the average classification accuracy. The present invention introduces the knowledge of the synthesis model through the knowledge distillation technology, which can effectively enhance the plasticity of the classification model, and reasonably balance the model stability and plasticity, and improve the average classification accuracy.

[0046] Compared with the prior art, the advantages and positive effects of the present invention are as follows: (1) Aiming at the problem of insufficient plasticity of the current synthesis replay online class-incremental continuous learning, the present invention uses the knowledge distillation technology to introduce the model knowledge of the synthesis model, which can effectively enhance the plasticity of the classification model; (2) The method provided by the present invention can better balance the plasticity and stability of the classification model by adjusting the weight of the distillation term, and improve the final average classification accuracy of the classification model; (3) The present invention makes the buffer able to store a larger number of images by downsampling the images in the buffer, improving the utilization efficiency of the buffer; (4) The method provided by the present invention can reduce the overfitting of the continuous learning model to the memory samples in the buffer; (5) The method provided by the present invention does not add any additional network parameters to the existing synthesis replay online class-incremental learning method, and is simple to operate and easy to train. Brief Description of the Drawings

[0047] Figure 1 is a flowchart of the synthetic replay online class-incremental continual learning method for introducing model knowledge provided by the present invention;

[0048] Figure 2 is a schematic structural diagram of the synthetic replay online class-incremental continual learning method for introducing model knowledge provided by the present invention; Detailed Description of the Embodiments

[0049] The present invention will be further described in conjunction with the embodiments and the drawings:

[0050] In order to enable those skilled in the art of this technology to better understand the solution of this application, the technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments and the drawings in this application. It should be understood that the drawings are only for illustrative purposes and cannot be construed as a limitation of this patent. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of this application.

[0051] Referring to "embodiments" in this application means that the specific features, structures, or characteristics described in conjunction with the embodiments may be included in at least one embodiment of this application. The appearance of this phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described in this application can be combined with other embodiments.

[0052] Figure 1 is a flowchart of the synthetic replay online class-incremental continual learning method for introducing model knowledge provided by the embodiments of the present invention, mainly including the following steps:

[0053] I. Construct a dynamic buffer:

[0054] To avoid the memory growth of the buffer, a buffer area with a fixed total memory size is opened up to store synthetic samples and real samples. When learning a new task, the buffer will receive new task samples in each batch, downsample them by s times and store them. This can enable the buffer with limited memory to store more samples for replay. If the buffer is full, when new task samples are stored in the buffer, real samples in the buffer will be randomly replaced. During this process, a fixed number of real samples of the new task in the buffer are initialized as synthetic samples, and these synthetic samples will be updated using the dataset distillation method and will not be replaced. To ensure the balance of the number of synthetic samples of each class in the buffer, the number of synthetic samples of each class is the same when initialized.

[0055] In the embodiment, the memory size of the buffer is set to a fixed value v. Let the total number of tasks for continuous learning be N, and the task sequence be: T1,..., T i ,...., T N . During the learning process of the new task T i , the buffer continuously receives new task sample batches, downsamples them, and then stores them. The purpose of the downsampling process is to enable the buffer with limited memory to store a larger number of samples for replay, promoting the model to retain more old knowledge. The downsampling process uses bicubic interpolation, and the pixel value calculation formula for the interpolation points is:

[0056]

[0057] where x and y are the coordinate values of the interpolation points, normalized to the range [0, 1], P(x, y) represents the pixel value of the interpolation point, a ij is a coefficient calculated from the pixel values and derivatives of 16 known points around the interpolation point, x i y j is the polynomial basis function, representing the product of the i-th power of x and the j-th power of y.

[0058] If the buffer is full, the real samples stored in it will be randomly replaced by new task samples to ensure that there are more categories of real samples in the buffer.

[0059] During this process, s*v / n images of each category of the new task in the buffer are initialized as synthetic samples. Where s is the image downsampling factor, and n is the total number of categories of all incremental tasks, that is:

[0060]

[0061] where n i is the number of categories of task T i . Task T i has a total of n i *s*v / n images initialized as buffer synthetic samples. The synthetic samples will not be replaced, but will be parameterized and updated using the dataset distillation method. After the initialization of the synthetic samples of the last task is completed, all samples in the buffer are synthetic samples, and the number of synthetic samples of each category is equal.

[0062] II. Dataset Distillation:

[0063] The method for updating synthetic samples uses dataset distillation to enable the synthetic model to have a similar training effect on a small number of synthetic samples and the entire original image dataset. After the classification model is trained on a small number of synthetic samples of old tasks, it can be similar to being trained on a large number of original images of old tasks, so as to better retain the knowledge learned before and avoid catastrophic forgetting. Since the buffer synthetic samples are downsampled by a factor of s to save memory, the real samples also need to be downsampled by a factor of s before dataset distillation. The present invention uses the method of gradient-matching dataset distillation to update the synthetic samples by minimizing the distance between the gradients of the synthetic model on the synthetic samples and the real samples. At the same time, a relational matching loss is added to make the synthetic samples and real samples of the new task category have a consistent relationship with the synthetic samples of the old tasks. During the dataset distillation process, the update of the synthetic model parameters and the synthetic samples alternates. Whenever a batch of new task samples arrives, the synthetic model will perform a parameter update. The synthetic samples are updated once every κ times the synthetic model parameter is updated. When the synthetic model parameters are updated, only the real samples and new task samples in the buffer are received, and the synthetic samples in the buffer are not received, so as to avoid strong coupling between the synthetic model parameters and the synthetic samples and affect the effect of dataset distillation.

[0064] Before dataset distillation for each task, the output dimension of the classification head of the synthetic model becomes the number of classes of all classes seen before plus the number of classes of this task, and the synthetic model parameters are re-initialized.

[0065] The downsampled real samples and synthetic samples need to be data-augmented, and the data augmentation includes a combination of randomly adjusting brightness, randomly adjusting saturation, randomly adjusting contrast, and random cropping.

[0066] In the embodiment, gradient-matching dataset distillation is used to update the synthetic samples.

[0067] Before dataset distillation for each task, the synthetic model needs to be redefined. The output dimension of the classification head of the synthetic model becomes the number of classes of all classes seen before plus the number of classes of this task, and the synthetic model parameters are re-initialized. When updating task T i In the process of synthetic samples, the synthetic samples need to be initialized first. If the buffer memory size is v and the total number of classes is n, then s*v / n pictures of each class in the buffer are used as the initialization of the synthetic samples, where s is the image downsampling factor. Then, in task T i A real sample set is collected in the data stream. The real samples are downsampled by a factor of s to make their resolution the same as that of the synthetic samples, and the data knowledge of this real sample set is distilled into the synthetic samples. Before calculating the dataset distillation loss, the real samples and synthetic samples need to be data-augmented, including randomly adjusting brightness, randomly adjusting saturation, randomly adjusting contrast, and random cropping.

[0068] The gradient matching loss for dataset distillation is as follows:

[0069]

[0070] where ψ is the parameter of the synthesis model, represents the set of downsampled real samples of class c when updating the synthetic samples for the j - th time, is the set of synthetic samples of class c in the buffer, and dist represents the distance metric, taking the L2 norm.

[0071] To achieve a better dataset distillation effect, a relationship matching loss is added on the basis of the gradient matching loss, so that the average features of the synthetic samples and the real samples have the same relationship as the average features of the synthetic samples of the old task, thereby making the synthetic data and the real data have a similar distribution.

[0072] The relationship matching loss is defined as:

[0073]

[0074] where M o represents all the synthetic samples of the old task, represents the average feature of the synthetic samples of class c obtained by the synthesis model, represents the average feature of all the synthetic samples of the old task obtained by the synthesis model, represents the average feature of the downsampled real samples of class c obtained by the synthesis model.

[0075] The loss of dataset distillation is obtained as:

[0076] L dd = L g + αL r

[0077] where α is the weight of the relationship matching loss.

[0078] The update method of the synthetic samples is:

[0079]

[0080] where η is the learning rate for updating the synthetic samples.

[0081] During the dataset distillation process, the parameters of the synthesis model must also be updated. The purpose is to enable the synthesis model to calculate the gradients of the synthetic samples and the real samples at different parameter points during the entire training process, making the range of gradient matching larger, so as to obtain a better dataset distillation effect.

[0082] During the learning process of task T i whenever there is a batch of new task samples B TiWhen a new task sample arrives, the synthesis model performs a parameter update. Every κ times the synthesis model updates its parameters, a synthetic sample is updated. When the synthesis model updates its parameters, it receives the new task sample B Ti and the real sample B in the randomly sampled buffer Mr for training, and the loss uses cross-entropy loss. Using the real sample B in the buffer Mr is to make the class information more sufficient when the gradients are matched, further improving the dataset distillation effect. The synthesis model is updated using real samples instead of synthetic samples because this will cause the synthetic samples to be strongly coupled with the synthesis model parameters, deteriorating the dataset distillation effect. At the same time, if the synthesis model is updated on synthetic samples, since the number of synthetic samples is very small, the synthesis model is also prone to overfitting.

[0083] III. Training the classification model:

[0084] In each task, the training process of the classification model is divided into two stages. In the first stage, to retain the knowledge of the old tasks, a portion of the samples are sampled from the buffer, upsampled by s times, and restored to the original image size. Then the buffer samples are merged with the new task sample batches, and further augmented using data augmentation to obtain the training input. The training loss uses cross-entropy loss, contrastive learning loss, and distillation loss. Among them, the distillation loss introduces the knowledge of the synthesis model through the method of soft-label knowledge distillation, which can enhance the ability to regulate the plasticity of the classification model. And since the synthesis model has not been updated on synthetic samples, the knowledge of the synthesis model can also help alleviate the overfitting of the classification model to synthetic samples. After the new task samples are learned, the training enters the second stage, where the classification model learns all the samples in the buffer, and the training loss is the same as that in the first stage. By learning the buffer samples, the classification model further learns the knowledge of new and old tasks, improving the classification accuracy.

[0085] In the first training stage of the classification model, when a new batch of task samples arrives, a fixed number of buffer samples are sampled and added to the training. If the number of samples in the buffer is insufficient, the classification model is not trained. Wait until the number of samples in the buffer reaches this fixed value, and then the classification model is trained.

[0086] The second training stage of the classification model requires data augmentation for the buffer samples, which is the same as the first stage. The data augmentation method is a combination of random cropping, random horizontal flipping, randomly adjusting the brightness, contrast, saturation, and hue of the image, and randomly converting to grayscale.

[0087] In the embodiment, the classification model can be structurally divided into a feature extractor f θ 、a projection layer and an output layer g ω . Among them, θ、 and ω represent the parameters of the feature extractor, projection layer, and output layer respectively. The feature extractor extracts the features of the input image to obtain f θ (x), and then passes through the projection layer and output layer respectively to obtain the projection and the output g ω (f θ (x)), where the projection is used to generate the contrastive learning loss, and the output g ω (f θ (x)) is the predicted label, which is used to generate the cross-entropy loss.

[0088] For task T i , the training of the classification model is divided into two stages. In the first stage, new task samples and samples in the buffer are learned. Specifically, whenever a batch of new task samples B Ti arrives, a part of the samples are randomly sampled from the buffer and upsampled s times using bicubic interpolation to restore them to the original image size, obtaining B M , and then merged with the new task samples to obtain the union B M ∪ B Ti . Then, data augmentation operations are used for expansion. Data augmentation includes random cropping, random horizontal flipping, random adjustment of the brightness, contrast, saturation, and hue of the image, and random conversion to grayscale. The final training input of the classification model is:

[0089] B = B M ∪ B Ti ∪ Aug(B M ∪ B Ti )

[0090] where Aug(·) represents the data augmentation operation.

[0091] At the initial stage of task T1, since there are no samples in the buffer, the classification model is not trained temporarily. After the number of samples in the buffer reaches |B M |, the classification model starts to be trained. This can avoid the change in the number of input samples of the classification model, increase the stability of the training process, and improve the classification accuracy.

[0092] The loss of the classification model is divided into contrastive learning loss L contrastive , cross-entropy loss L ce and knowledge distillation loss L kd :

[0093] L = L contrastive + L ce + λL kd

[0094] where L contrastive is the contrastive learning loss, L ceis the cross-entropy loss, L kd is the knowledge distillation loss, λ represents the coefficient of the distillation term, and the larger λ is, the more knowledge the classification model receives from the synthetic model. By adjusting the value of λ, the trade-off between the stability and plasticity of the classification model can be made better.

[0095] The formula for the contrastive learning loss is:

[0096]

[0097] where z i represents the projection of the i-th sample in B, P(i) represents the set of positive samples of sample i in B, that is, the samples with the same class label as sample i, A(i) represents the set of samples in B except sample i, and τ represents the temperature coefficient.

[0098] The formula for the cross-entropy loss is:

[0099]

[0100] where y i represents the one-hot encoding of sample i, and q i represents the predicted label of sample i.

[0101] The knowledge distillation loss introduces the knowledge of the synthetic model, which improves the plasticity of the classification model. Changing the weight of this distillation term can adjust the plasticity and stability of the model, making the model achieve a better trade-off between stability and plasticity. At the same time, since the synthetic model has not learned the synthetic samples in the buffer, the knowledge distillation of the synthetic model on the classification model can also alleviate the overfitting of the classification model to the synthetic samples.

[0102] The formula for the knowledge distillation loss is:

[0103]

[0104] where ο i represents the predicted label obtained by the synthetic model when inputting sample i.

[0105] The final loss function is L = L contrastive + L ce + λL kd , where λ represents the coefficient of the distillation term, and the larger λ is, the more knowledge the classification model receives from the synthetic model. By adjusting the value of λ, the trade-off between the stability and plasticity of the classification model can be made better.

[0106] After obtaining the loss of the classification model, the loss is backpropagated to obtain the gradient, and the parameters of the classification model are updated using the mini-batch stochastic gradient descent method.

[0107] In an online continuous learning setting, the model can only access the data stream once. After the classification model has learned the data samples of task T i , it can no longer access the learned data stream. At this time, it enters the training process of the second stage, and uses all the samples in the buffer as the training input for training.

[0108] All the samples in the buffer are divided into data batches, and the classification model is trained using the method of batch stochastic gradient descent. For each batch of buffer samples, first use bicubic interpolation to upsample them by s times to restore them to the original image size, and then use the same data augmentation operation as in the first training stage for expansion to obtain the training input. The loss function in this stage is the same as that in the first stage, but the gradient is clipped to 1 / 10 to avoid overfitting of the classification model on the buffer samples. After learning in the second stage, the classification model can further learn the knowledge of new tasks and old tasks, and improve the classification accuracy.

[0109] IV. Testing the classification accuracy of the classification model:

[0110] After training each incremental task, the classification model is tested. The test samples include all classes of the incremental tasks that the model has seen. During testing, first upsample the buffer samples by s times, then extract the features of all samples in the buffer by the feature extractor of the classification model, then obtain the mean feature of all classes by class, and finally calculate the features of the test samples, and classify the test samples into the class with the mean feature closest to its feature. The performance of the continuous learning model is usually evaluated by the average accuracy. After statistically calculating the classification accuracy of each task, the average accuracy of the classification model can be calculated. After training all the incremental tasks, the average classification accuracy of all tasks can be obtained.

[0111] In the embodiment, the performance of the continuous learning model is evaluated by the average accuracy. The calculation formula of the average accuracy is as follows:

[0112]

[0113] where a k,j represents the classification accuracy of the continuous learning model on the j-th task after learning k tasks.

[0114] After learning all the data samples of task T i , test the classification accuracy of the classification model on tasks T1 to T i , and then obtain the average accuracy A i . First, use bicubic interpolation to upsample the samples in the buffer by s times, then calculate the features of all samples in the buffer by the feature extractor of the classification model, and calculate the feature mean of each class of samples. For each test sample x j , calculate its feature f j by the feature extractor, then xj The predicted label is:

[0115]

[0116] where f j is the feature of the test sample x calculated by the classification model feature extractor j , and F i is the feature mean of all samples of the i-th class in the buffer.

[0117] Let the number of incremental tasks be N. After the classification model finishes learning task T N , the final average precision A N is calculated.

[0118] In a specific embodiment of the present application, the above method is applied to the image datasets CIFAR-100 and Tiny-ImageNet widely used in the field of image classification. By comparing with the existing state-of-the-art synthetic replay online class-incremental continual learning method SSD, the effectiveness of the method proposed in the present application is demonstrated.

[0119] (1) Dataset introduction:

[0120] The CIFAR-100 dataset was released by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton of the University of Toronto in 2009 and contains 60,000 color images of 32×32 pixels. These images have 100 classes, with 600 images in each class, of which 500 are used for training and 100 are used for testing.

[0121] The Tiny-ImageNet dataset was extracted from the ImageNet dataset and released by Stanford University in 2016. It contains 120,000 color images of 64×64 pixels. These images have 200 classes, with 500 training images, 50 validation images, and 50 test images in each class respectively.

[0122] (2) Experimental settings:

[0123] Experiments were conducted on the cases of saving 1, 5, and 10 pictures for each class in the buffer. The CIFAR-100 dataset has 100 classes, which are divided into 10 incremental tasks, with 10 classes in each task, and experiments were conducted under the settings of buffer sizes of 100, 500, and 1000 respectively. The Tiny-ImageNet dataset has 200 classes, which are divided into 20 incremental tasks, with 10 classes in each task, and experiments were conducted under the settings of buffer sizes of 200, 1000, and 2000 respectively. For all experiments in this embodiment, the final results were obtained by running 3 repeated experiments and then taking the average.

[0124] (3) Hyperparameter Selection:

[0125] For CIFAR-100, the downsampling factor s of the buffer samples is set to 2, and for Tiny-ImageNet, the downsampling factor s of the buffer samples is set to 1. In the experiment, the output channel number of the original initial convolutional layer of ResNet18 is modified from 64 to 20 as the feature extractor of the classification model. The batch size |B Ti | of the new task samples is set to 10, and the number of buffer samples |B M | sampled each time is set to 100. In the loss function, the temperature coefficient τ of the contrastive learning loss is set to 0.07, and the weight λ of the knowledge distillation loss is set to 0.2. The classification model is updated using the SGD optimizer, with the learning rate set to 0.1 and the momentum set to 0.9. The synthesis model uses a 3-layer convolutional neural network. In the dataset distillation loss, the weight α of the relationship matching loss is set to 1, the learning rate η for updating the synthetic samples is set to 0.004, and the update interval κ of the synthetic samples is set to 6.

[0126] Table 1 shows the experimental results of training and testing the method of the present invention under various experimental settings on the CIFAR-100 and Tiny-ImageNet datasets. The results show that the method proposed by the present invention has a higher final average precision than SSD, demonstrating the effectiveness of the method proposed by the present invention.

[0127]

[0128] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A synthetic replay online class incremental continuous learning method for introducing model knowledge, characterized in that: The total number of tasks for continuous learning is N, and the task sequence is: T1,....,T i ,....,T N , and the incremental continuous learning steps are as follows: Construct a buffer with a memory size of v; for each batch of newly received tasks T i The samples are downsampled by a factor of s to obtain real samples. For each class, s*v / n real sample images are initialized to obtain synthetic samples, where s is the image downsampling factor and n is the total number of classes of all incremental tasks; the real samples and synthetic samples are stored in a buffer; Update the synthetic samples using the gradient-matching dataset distillation method to obtain the distilled synthetic samples; Randomly sample the real samples and distilled synthetic samples in the buffer, upsample the sampled samples by s times to obtain the sampled samples that restore the original image size, and combine this sample with the new task T i of this batch of samples to obtain an enhanced and augmented data sample; input the enhanced and augmented data sample into the classification model to train the classification model and complete the incremental task training of this batch. After training the new task T i For all batches of the classification model, test the classification model; After all incremental tasks are trained, aggregate the classification accuracies of all tasks and then average them to obtain the average classification accuracy for all tasks.

2. The synthetic replay online class incremental continuous learning method for introducing model knowledge according to claim 1, wherein: The downsampling by a factor of s uses bicubic interpolation. The formula for calculating the pixel value of the interpolation point in bicubic interpolation is: where x and y are the coordinate values of the interpolation points, normalized to the range [0, 1], P(x, y) represents the pixel value of the interpolation point, and a ij is a coefficient calculated from the pixel values and derivatives of 16 known points around the interpolation point, and x i y j is a polynomial basis function, representing the product of the i-th power of x and the j-th power of y.

3. The method for synthetic replay online class incremental continuous learning for introducing model knowledge according to claim 1, characterized in that: When the buffer receives a new batch of task samples, if the memory is full, the real samples in it will be randomly replaced by the new task samples, while the synthetic samples are updated after initialization and are not replaced.

4. The method for synthetic replay online class incremental continuous learning that introduces model knowledge according to claim 1, characterized in that: The gradient-matching dataset distillation method mentioned above is: Update the synthetic samples by minimizing the distance between the gradients of the synthetic model on the synthetic samples and the real samples. On the basis of the original loss, add the relationship matching loss to make the synthetic samples and real samples of the new task categories have a consistent relationship with the synthetic samples of the old tasks; during the dataset distillation process, the update of the synthetic model parameters and the synthetic samples alternates. Whenever a batch of new task samples arrives, the synthetic model will perform a parameter update. After the synthetic model undergoes κ parameter updates, the synthetic samples are updated once; when the synthetic model parameters are updated, only the real samples and new task samples in the buffer are received, and the synthetic samples in the buffer are not received to avoid strong coupling between the synthetic model parameters and the synthetic samples.

5. The method for synthetic replay online class incremental continuous learning for introducing model knowledge according to claim 1 or 4, characterized in that: During dataset distillation, before distilling each task dataset, the output dimension of the classification head of the synthetic model becomes the number of classes of all classes seen before plus the number of classes of this task, and the synthetic model parameters are re-initialized.

6. The method for synthetic replay online class incremental continuous learning that introduces model knowledge according to claim 4, characterized in that: The relationship matching loss formula is: L r = dist(dist(h Mc , h Mo ), dist(h Dc , h Mo )) where h Mc represents the average feature of the synthetic samples of class c obtained by the synthetic model, and h Dc represents the average feature of the downsampled real samples of class c obtained by the synthetic model, and h Mo represents the average feature of all synthetic samples of the old tasks obtained by the synthetic model, and dist represents the distance metric, taking the L2 norm; The final dataset distillation loss is: L dd = L g + αL r where L g is the gradient matching loss, and α is the weight of the relation matching loss.

7. The method for synthetic replay online class incremental continuous learning for introducing model knowledge according to claim 1, characterized in that: The training of the classification model is divided into two stages; in the first stage, to retain the knowledge of the old tasks, sample some samples from the buffer, upsample them by a factor of s to restore to the original image size, then merge the buffer samples with the new task sample batch, and then use data augmentation to expand, thus obtaining the training input; the training loss of the classification model includes cross-entropy loss, contrastive learning loss, and knowledge distillation loss; among them, the knowledge distillation loss introduces the knowledge of the synthetic model through the method of soft-label knowledge distillation to enhance the control ability of the plasticity of the classification model, and since the synthetic model has not been updated on the synthetic samples, the knowledge of the synthetic model can also help alleviate the overfitting of the classification model to the synthetic samples; after the new task samples are learned, enter the second stage of training, and the classification model learns all the samples in the buffer. By learning the buffer samples, the classification model further learns the knowledge of the new tasks and the old tasks and improves the classification accuracy.

8. The method for synthetic replay online class incremental continuous learning that introduces model knowledge according to claim 5, wherein: When the classification model is trained, if the number of samples in the buffer is insufficient, the classification model training will not be carried out.

9. The method for synthetic replay online class incremental continuous learning for introducing model knowledge according to claim 4, wherein: The training loss of the classification model is: L = L contrastive + L ce + λL kd where \(L\) contrastive is the contrastive learning loss, \(L\) ce is the cross-entropy loss, \(L\) kd is the knowledge distillation loss, and \(\lambda\) represents the coefficient of the distillation term; The contrastive learning loss formula is: where B represents the training input of the classification model, and z i represents the projection of sample i in B generated by the classification model, P(i) represents the set of positive samples of sample i in B, that is, the samples with the same class as sample i, A(i) represents the set of samples in B except sample i, and τ represents the temperature coefficient; The cross-entropy loss is: where y i represents the one-hot encoding of sample i, and q i represents the predicted label of sample i; The knowledge distillation loss formula is: where B represents the training input of the classification model, q i represents the predicted label of the classification model for sample i, ο i represents the predicted label of the synthesis model for sample i.

10. The method for synthetic replay online class incremental continuous learning for introducing model knowledge according to claim 1, characterized in that: When testing the classification model, first upsample the samples in the buffer by s times, then extract the features of all samples in the buffer by the feature extractor of the classification model, then obtain the mean features of all classes by class, and finally calculate the features of the test samples, and classify the test samples into the class with the mean feature closest to its feature; The performance of the continual learning model is usually evaluated by the average precision. After statistically calculating the classification accuracy of each task, the average precision of the classification model is obtained; the test samples include all classes of the incremental tasks seen by the model.

Images