A continual learning image classification method based on generative replay

Abstract

This invention provides a continuous learning image classification method based on generative playback, relating to the fields of continuous learning and computer vision. This method uses an image classification network to continuously learn the image classification task; it uses an image generation network to learn the distribution of images for the current task, and when a new task arrives, it generates images of the learned task as playback data, mitigating the catastrophic forgetting problem of the image classification network during continuous learning. The method employs a Teacher-Student architecture, using the image classification network for the (t-1)th task as the Teacher model, and initializing task T with its parameters. t An image classification network is used as the Student model; the Teacher model remains fixed during subsequent training to assist the Student model in remembering old knowledge. When a new task arrives, an image generation network generates pseudo-data from past tasks, enabling the model to distinguish between old and new task knowledge when learning a new task.

Description

Technical Field

[0001] This invention relates to the fields of continuous learning and computer vision technology, and in particular to a continuous learning image classification method based on generative playback. Background Technology

[0002] Current deep learning-based image classification methods are mostly data-driven, requiring not only large datasets for training but also heavily relying on fixed datasets in a fixed environment. When new data appears, due to the connectivity characteristics of neural networks, the model suffers from "catastrophic forgetting," meaning it catastrophically forgets old knowledge after learning new knowledge. For the model to master this "new knowledge," it needs to be retrained using all the datasets. Continuous learning methods, on the other hand, can adaptively solve new tasks using only new task data when historical task data is inaccessible, while retaining previously learned knowledge about old tasks. This facilitates efficient agent training. However, continuous learning also suffers from the catastrophic forgetting problem. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a continuous learning image classification method based on generative playback. Through generative playback, pseudo-data of historical tasks is generated and played back to the model, enabling the model to remember old knowledge, thereby alleviating the problem of catastrophic forgetting and achieving image classification.

[0004] To address the aforementioned technical problems, the present invention provides a continuous learning image classification method based on generative playback. This method uses an image classification network to continuously learn the image classification task and uses an image generation network to learn the distribution of images for the current task. When a new task arrives, images of the learned task are generated as playback data, thus mitigating the catastrophic forgetting problem of the image classification network during continuous learning.

[0005] The form of an image classification network is =C(x;θ,V)=H(F(x;θ);V), where C represents the image classification network, u=F(x;θ) is the feature extractor, and θ is the parameter of the feature extraction part. For the classifier, matrix V projects the output u of the feature extractor onto the class scores, and then normalizes them to class probabilities using the softmax function. ;

[0006] Let T be the number of tasks that arrive in the continuous learning of image classification. t The corresponding dataset is x i It is the i-th image. It is the label corresponding to the i-th image. It is the vocabulary corresponding to the image categories; then for tasks T other than the first task... t Continuous learning for image classification includes the following steps:

[0007] Step 1: Using the Teacher-Student architecture, the image classification network C for the (t-1)th task... t-1 (θ t-1 ) as the Teacher model, and using its parameters θ t-1 Initialize task T t Image classification network C t The Teacher model is used to assist the training of the Student model, enabling the Student model to remember old knowledge.

[0008] Step 2: Generate network M from the image of task t-1. t-1 Generating pseudo data for the first t-1 tasks , It is an image generation network M t-1 The generated image, It is to generate image labels. Includes all classes of the first t-1 tasks, z i It is a random latent vector sampled from a simple distribution, N R It represents the number of images played back from previous tasks;

[0009] Step 3: Train the image classification network;

[0010] Step 3.1: Define a difficulty predictor P to determine the difficulty or complexity of each sample; the smaller the predicted value, the simpler the sample. Use the difficulty predictor on the task dataset. Make a prediction to obtain each image sample x in the dataset. i Difficulty score r i As shown in the formula below:

[0011] ;

[0012] Step 3.2: Define a training scheduler f(λ)∈(0,1], where λ is a hyperparameter controlling the learning speed; determine the proportion of image samples selected in each training epoch using f(λ), and select the simplest f(λ)*num(λ) based on the difficulty score r obtained in Step 3.1. The task dataset consists of 10 image samples. As the number of iterations increases, f(λ) gradually increases until f(λ) = 1.

[0013] Step 3.3: Transfer the current task data Input image classification network Ct (θ t During training, the loss of an image classification network consists of two parts: classification loss and classification loss. and distillation loss As shown in the formula below:

[0014] ;

[0015] ;

[0016] Among them, y i It is image x i The one-hot vector representation of the real label. It is a classification network for image x i Predicted labels, Let l be the classification loss function and l be the distillation loss function. for The quantity, F t-1 and F t Representing task T respectively t-1 and T t Feature extractor for image classification networks;

[0017] Step 4: Train the image generation network;

[0018] Step 4.1: Retain the image generation network M for the (t-1)th task. t-1 (Φ t-1 ) as the Teacher model, and using its parameter Φ t-1 Initialize task T t Image generation network M t , as the Student model;

[0019] Step 4.2: The loss of the image generation network consists of two parts: one part is the loss of the image generation task. Part of the loss is due to playback. The playback loss causes the image generation network M to... t and M t-1 To produce similar outputs under the same input, the formula is as follows:

[0020] ;

[0021] ;

[0022] in, The loss function, the form of which is determined by the generator chosen by the generator network, for The L2 norm;

[0023] The total loss of the image generation network is then... ;

[0024] Step 5: Read the test image x and input it into the image classification network C. t In this process, the probability value of the test image for each possible category is obtained. =C t (x;θ,V)=H t (F t (x;θ);V), the position of the maximum probability of the category j=max( In the vocabulary list The corresponding value in This is the predicted category of image x.

[0025] For the first task T1, the process of continuously learning image classification is similar to that of other tasks T1 besides the first task. t The process is the same, except that in steps 1 and 4.1, the image classification network and the image generation network are trained from the random initialization network. In step 3.1, the difficulty prediction score is defined as a random probability value. The training process of the image classification network does not require the use of the playback image generated in step 2, and the training process of the image generation network does not require the playback loss.

[0026] The beneficial effects of adopting the above technical solution are as follows: The continuous learning image classification method based on generative playback provided by the present invention generates pseudo data of historical tasks using an image generation network when a new task arrives, so that the model can acquire the ability to distinguish between new and old task knowledge when learning a new task, effectively alleviating the catastrophic forgetting problem of image classification networks in the continuous learning process. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the image classification network training process for the t-th task provided in an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the image generation network training process for the t-th task provided in an embodiment of the present invention;

[0029] Figure 3 The images provided in this embodiment of the invention are real images and replay images corresponding to different tasks, wherein (a) is the real image of task 1, (b) is the real image of task 2, (c) is the real image of task 3, (d) is the real image of task 4, (e) is the real image of task 5, (f) is the replay image of task 2, (g) is the replay image of task 3, (h) is the replay image of task 4, and (i) is the replay image of task 5.

[0030] Figure 4This is a graph showing the average accuracy of the image classification method on the test set after learning each task, as provided in this embodiment of the invention. Detailed Implementation

[0031] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0032] This embodiment designs a Disjunction MNIST experiment on the MNIST dataset of handwritten digit images, and it is a classification task with incremental category distribution. The dataset is divided into five disjoint tasks, each containing two classes of digits, corresponding to: Task 1 = {0, 1}, Task 2 = {2, 3}, Task 3 = {4, 5}, Task 4 = {6, 7}, and Task 5 = {8, 9}. When a new task arrives, the image classification method needs to learn to distinguish all the data categories encountered.

[0033] In this embodiment, a continuous learning image classification method based on generative playback is proposed. The method uses an image classification network to continuously learn the image classification task; it uses an image generation network to learn the distribution of the current task images; and when a new task arrives, it generates images of the learned tasks as playback data, thereby alleviating the catastrophic forgetting problem of the image classification network in the continuous learning process.

[0034] In this embodiment, the image classification network C is implemented using a ResNet18 network for continuous learning of image classification tasks. The image generation network uses a Deep Convolutional Generative Adversarial Network (DCGAN) and is improved to a GAN network that uses class labels as auxiliary generation conditions. This GAN network consists of two parts: a generator G and a discriminator D.

[0035] In this embodiment, the image classification network takes the form of: =C(x;θ,V)=H(F(x;θ);V), where C represents the image classification network, u=F(x;θ) is the feature extractor, and θ is the parameter of the feature extraction part. For the classifier, matrix V projects the output u of the feature extractor onto the class scores, and then normalizes them to class probabilities using the softmax function. ;

[0036] Let T be the number of tasks that arrive in the continuous learning of image classification. t The corresponding dataset is x i It is the i-th image. It is the label corresponding to the i-th image. It is the vocabulary corresponding to the image categories; then for tasks T other than the first task... tContinuous learning for image classification includes the following steps:

[0037] Step 1: Using the Teacher-Student architecture, the image classification network C for the (t-1)th task... t-1 (θ t-1 ) as the Teacher model, and using its parameters θ t-1 Initialize task T t Image classification network C t The Teacher model remains fixed during subsequent training and is used to assist the Student model in training so that the Student model can remember old knowledge.

[0038] Step 2: Generate network G from the image of task t-1. t-1 Generating pseudo data for the first t-1 tasks , It is an image generation network M t-1 The generated image, It is to generate image labels. Includes all classes of the first t-1 tasks, z i It is a random latent vector sampled from a simple distribution, N R It represents the number of images played back from previous tasks;

[0039] Step 3: Train the image classification network, such as... Figure 1 As shown;

[0040] Step 3.1: Define a difficulty predictor P to determine the difficulty or complexity of each sample; the smaller the predicted value, the simpler the sample. Use the difficulty predictor on the task dataset. Make a prediction to obtain each image sample x in the dataset. i Difficulty score r i As shown in the formula below:

[0041] ;

[0042] In this embodiment, the classification network C for the (t-1)th task is... t-1 (θ t-1 As a difficulty predictor, , y i It is image x i The one-hot vector representation of the real label. Let be the classification loss function for the image classification network.

[0043] Step 3.2: Define a training scheduler f(λ)∈(0,1], where λ is a hyperparameter controlling the learning speed; determine the proportion of image samples selected in each training epoch using f(λ), and select the simplest f(λ)*num(λ) based on the difficulty score r obtained in Step 3.1. The dataset consists of 10 image samples. As the number of iterations increases, f(λ) gradually increases until f(λ) = 1.

[0044] Step 3.3: Transfer the current task data Input image classification network C t (θ t During training, the loss of the image classification network in this embodiment consists of two parts: classification loss. and distillation loss The loss function is as follows:

[0045] ;

[0046] ;

[0047] Among them, y i It is image x i The one-hot vector representation of the real label. It is a classification network for image x i Predicted labels, Let l be the classification loss function and l be the distillation loss function. for The quantity, F t-1 and F t Representing task T respectively t-1 and T t A feature extractor for an image classification network; during training, the loss of the image classification network is minimized. .

[0048] In this embodiment, the classification loss uses cross-entropy loss, i.e. Distillation loss is calculated using L2 loss, i.e. ;

[0049] Step 4: Train the image generation network M, such as... Figure 2 As shown in the figure, the two dashed boxes represent the training processes of the discriminator D and the generator G, respectively.

[0050] Step 4.1: Retain the generator G for the (t-1)th task. t-1 (Φ t-1 ) as the Teacher model, and using its parameter Φ t-1 Initialize task T t Generator G t;

[0051] Step 4.2: Alternately update discriminator D t and generator G t With the discriminator D t The generator G is updated every n_critic = 5 epochs. t In each iteration, D t and G t The training process is as follows:

[0052] Step 4.2.1: From the current generator G t Generate pseudo-data for the categories contained in the t-th task. , , This refers to the image labels to be generated, containing all classes of the t-th task. For the discriminator, the current task data... If true, the class label is 1; the generated pseudo data If the result is false, the class label is 0. The dataset is then obtained. .

[0053] Step 4.2.2: Train the discriminator; [The dataset is then processed / processed]. The input discriminator D, in this embodiment the discriminator loss is the binary cross-entropy loss. The formula is as follows:

[0054] ;

[0055] in, , , representing the discriminator's prediction score for the image.

[0056] Step 4.2.3: Train the generator G t (Φ t The generator's loss consists of two parts: one part is the loss from the image generation task. Part of it is the playback alignment loss. Playback alignment loss causes the image generation network G to... t and G t-1 To produce similar outputs under the same input, in order to maintain the generator's ability to generate old task data, the formula is as follows:

[0057] ;

[0058] ;

[0059] in, for The L2 norm;

[0060] The total loss of generator G is:

[0061] ;

[0062] Step 4.3: Determine the total loss of the image network;

[0063] The total loss of the image generation network is ;

[0064] Step 5: Read the test image x and input it into the image classification network C. t In this process, the probability value of the test image for each possible category is obtained. =C t (x;θ,V)=H t (F t (x;θ);V), the position of the maximum probability of the category j=max( In the vocabulary list The corresponding value in This is the predicted category of image x.

[0065] For the first task T1, the process of continuously learning image classification is similar to that of other tasks T1 besides the first task. t The process is the same, except that in steps 1 and 4.1, the image classification network and the image generation network are trained starting from random initialization. In step 3.1, the difficulty prediction score is defined as a random probability value. The training process of the image classification network does not require the use of the playback image generated in step 2, and the training process of the image generation network does not require playback loss. That is, for the first task T1, the loss functions of the image classification network and the image generation network are:

[0066] Loss function for image classification networks: ;

[0067] Loss function of image generation network: ;

[0068] In this embodiment, the composition of the dataset during the training process of each task is as follows: Figure 3 As shown in the figure. The first row of images represents the real images obtainable for the current task, and the second row represents the replay data generated by the generator. The first task is a binary classification task of {0, 1}, which does not require replay data. The second task is a binary classification task of {2, 3}, which requires generating replay data for categories {0, 1}. At this point, the image classification is a four-category task of {0, 1, 2, 3}; tasks 3, 4, and 5 follow the same pattern. In this embodiment, the average accuracy of the image classification method on the test set after learning each task is as follows: Figure 4As shown in the figure, the x-axis represents the number of categories that can be classified at the current time and the corresponding categories, while the y-axis represents the average accuracy on the corresponding test set. It can be seen that the image classification method maintains its performance during continuous learning, without experiencing catastrophic forgetting.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A continuous learning image classification method based on generative playback, characterized in that: Image classification networks are used to continuously learn image classification tasks; image generation networks are used to learn the distribution of images for the current task and generate images of the learned tasks as replay data when a new task arrives, thus mitigating the catastrophic forgetting problem of image classification networks during continuous learning. The image classification network takes the form of: =C(x;θ,V)=H(F(x;θ);V), where C represents the image classification network, u=F(x;θ) is the feature extractor, and θ is the parameter of the feature extraction part. For the classifier, matrix V projects the output u of the feature extractor onto the class scores, and then normalizes them to class probabilities using the softmax function. ; The method sets the condition for each incoming task T in the continuous learning image classification process. t The corresponding dataset is x i It is the i-th image. It is the label corresponding to the i-th image. It is the vocabulary corresponding to the image categories; then for tasks T other than the first task... t Continuous learning for image classification includes the following steps: Step 1: Using the Teacher-Student architecture, the image classification network C for the (t-1)th task is... t-1 (θ t-1 ) as the Teacher model, and using its parameters θ t-1 Initialize task T t Image classification network C t The Teacher model is used to assist in the training of the Student model, enabling the Student model to remember old knowledge. Step 2: Generate network M from the image of task t-1. t-1 Generating pseudo data for the first t-1 tasks , It is an image generation network M t-1 The generated image, It is to generate image labels. Includes all classes of the first t-1 tasks, z i It is a random latent vector sampled from a simple distribution, N R It represents the number of images played back from previous tasks; Step 3: Train the image classification network; Step 3.1: Define a difficulty predictor P to determine the difficulty or complexity of each sample; the smaller the predicted value, the simpler the sample. Use the difficulty predictor on the task dataset. Make a prediction to obtain each image sample x in the dataset. i Difficulty score r i The formula is shown below: ； The classification network C for the (t-1)th task t-1 (θ t-1 As a difficulty predictor, , y i It is image x i The one-hot vector representation of the real label. Let be the classification loss function for the image classification network; Step 3.2: Define a training scheduler f(λ)∈(0,1], where λ is a hyperparameter controlling the learning speed; determine the proportion of image samples selected in each training epoch using f(λ), and select the simplest f(λ)*num(λ) based on the difficulty score r obtained in Step 3.

1. The task dataset consists of 10 image samples. As the number of iterations increases, f(λ) gradually increases until f(λ) = 1. Step 3.3: Transfer the current task data Input image classification network C t (θ t During training, the loss of an image classification network consists of two parts: classification loss and classification loss. and distillation loss As shown in the formula below: ； ； Among them, y i It is image x i The one-hot vector representation of the real label. It is a classification network for image x i Predicted labels, Let l be the classification loss function and l be the distillation loss function. for The quantity, F t-1 and F t Representing task T respectively t-1 and T t Feature extractor for image classification networks; Step 4: Train the image generation network; Step 5: Read the test image x and input it into the image classification network C. t In this process, the probability value of the test image for each possible category is obtained. =C t (x;θ,V)=H t (F t (x;θ);V), the position of the maximum probability of the category j=max( In the vocabulary list The corresponding value in This is the predicted category of image x.

2. The continuous learning image classification method based on generative playback according to claim 1, characterized in that: The specific method for step 4 is as follows: Step 4.1: Retain the image generation network M for the (t-1)th task. t-1 (Φ t-1 ) as the Teacher model, and using its parameter Φ t-1 Initialize task T t Image generation network M t , as the Student model; Step 4.2: The loss of the image generation network consists of two parts: one part is the loss of the image generation task. Part of the loss is due to playback. The playback loss causes the image generation network M to... t and M t-1 To produce similar outputs under the same input, the formula is as follows: ； ； in, The loss function, the form of which is determined by the generator chosen by the generator network, for The L2 norm; The total loss of the image generation network is then... .

3. The continuous learning image classification method based on generative playback according to claim 2, characterized in that: For the first task T1, the process of continuously learning image classification is similar to that of other tasks T1 besides the first task. t The process is the same, except that in steps 1 and 4.1, the image classification network and the image generation network are trained from the random initialization network. In step 3.1, the difficulty prediction score is defined as a random probability value. The training process of the image classification network does not require the use of the playback image generated in step 2, and the training process of the image generation network does not require the playback loss.

Images