Method and device for solving catastrophic memory of forgotten learning based on knowledge distillation

Abstract

The invention belongs to the technical field of information data processing, and particularly relates to a method and device for solving catastrophic memory of learning forgetting based on knowledge distillation. The method comprises the steps that an initial model is selected, a standard data set is used for training, and initial model parameters are optimized by minimizing cross entropy loss; selecting a target forgetting category, adjusting model parameters by applying a category forgetting method, and generating a forgetting model which is low in target forgetting category identification capability and high in other category identification accuracy; and selecting residual category data from the original data set as incremental learning data, adding quantitative target forgetting category data as input data, taking the generated forgetting model as a teacher model, initializing a student model, constraining output distribution by the student model on the input data through knowledge distillation loss, and updating parameters of the student model. The problem that the forgotten model cannot keep the forgetting performance in the incremental learning process, and catastrophic forgetting may occur when new information is absorbed is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of information data processing, and more specifically, relates to a method and device for solving catastrophic recall caused by forgotten learning based on knowledge distillation. Background Art

[0002] The rapid development of artificial intelligence and machine learning technologies has significantly boosted social productivity. However, the widespread application of these technologies has also sparked widespread concern about data privacy and security. Machine learning models typically rely on large amounts of training data to improve performance, but this data may contain sensitive user information. This necessitates the ability to remove specific data knowledge without retraining, giving rise to research on "machine forgetting" techniques.

[0003] There have been many approximate forgetting methods in recent years. The PUMA method achieves the goal of approximate forgetting by simulating the impact of each training data point on model performance. The "amnesiac forgetting learning" method prompts the model training process to record which batches contain sensitive data and the corresponding parameter updates. If a data removal request occurs, the parameter updates will only be for the specified batches. However, existing research on forgetting methods only focuses on a single forgetting task, thus ignoring the consideration of subsequent incremental learning. In terms of data security, Wang et al. focused their discussion on the problem of data residue, that is, despite efforts to eliminate specific data traces, the model may continue to retain these traces. Liu et al. proposed a unified workflow involving machine forgetting, which includes three stages: training stage, forgetting learning stage and post-forgetting learning stage. The post-forgetting learning stage requires the model to provide reasoning services, and the subsequent behavior of the forgotten model has attracted increasing attention.

[0004] Chinese invention patent CN118298279A discloses a knowledge distillation method, system, medium and equipment for cross-domain passive domain data, including using a pre-trained teacher model to first train an image generator and then train an image encoder to preheat a student model at the same time, and using their respective loss functions to update parameters separately; using the image generator, image encoder and teacher model to train a mask network on a student domain training dataset, and then training the student model on the student domain training dataset, and adjusting the weights of domain-specific information loss and cross-domain knowledge distillation loss through stage factors; inputting the image to be predicted into the trained student model to obtain the classification label of the image to be predicted.

[0005] As can be seen, research progress in this field has been substantial in recent years, primarily focused on improving the efficiency of forgetting algorithms and the effectiveness of data removal. However, the issue of forgotten models failing to maintain their forgetting performance during incremental learning has been overlooked. Incremental learning involves the gradual learning of new tasks or categories while maintaining prior knowledge. However, existing research has not fully explored the performance of forgetting models after incremental learning. In fact, models can experience catastrophic recall when absorbing new information. Summary of the Invention

[0006] The present invention aims to overcome at least one of the defects of the above-mentioned prior art and provide a method for solving catastrophic recall of forgotten learning based on knowledge distillation to solve the problem of catastrophic recall, that is, the model may inadvertently relearn the features of forgotten categories during the incremental learning process, thereby causing the problem of forgetting failure.

[0007] The detailed technical solutions of the present invention are as follows: A method for solving catastrophic recall in forgotten learning based on knowledge distillation, the method comprising: S1. Select an initial model and train it using a standard dataset, optimizing the initial model parameters by minimizing the cross entropy loss. S2. Select the target forgetting category and adjust the model parameters using the forgetting method to generate a forgetting model whose recognition ability for the target forgetting category is lower than that for other categories; S3. Select the remaining category data except the target forgotten category from the original dataset as incremental learning data and add a certain amount of target forgotten category data as input data. Use the generated forgetting model as the teacher model to initialize the student model. The student model constrains the output distribution on the input data through knowledge distillation loss and updates the parameters of the student model.

[0008] Furthermore, the S1 specifically includes: S11. Select a standard image dataset and use it as training data; S12. Select a convolutional neural network architecture as the initial model ; S13. Train the initial model using each category of the dataset , so that its recognition ability for each category is improved. The training process optimizes the initial model parameters by minimizing the cross entropy loss LCE: (1); In formula (1), is the input data, is the label corresponding to the input data, is the total number of categories, Is the initial model For input data Belong to category The predicted probability of . Preferably, the label is one-hot encoded.

[0009] Furthermore, the S2 specifically includes: S21, select a target specific category from the data set as the forgotten target category, and then perform a forgetting operation on this category; S22. Simultaneously apply two forgetting methods: one is amnesia forgetting learning, and the other is KL divergence forgetting method; Amnesiac forgetting learning method: Record each mini-batch of samples and their corresponding parameter updates during training. When receiving a request to forget a specific category, backtrack the parameter updates of the batch that includes samples of the specific category, thereby removing the learning traces of the data of the forgotten target category. KL divergence forgetting method: select another data set, and select a sample category from this data set whose similarity with the forgotten target category is greater than the set threshold, and the number of sample categories is the same as the forgotten target category. Through fitting, that is, KL divergence, the sample category is aligned with the output distribution of the target category, so that the model's recognition ability of the forgotten target category is further reduced.

[0010] Furthermore, the amnesia forgetting learning specifically includes: (1) Record the training process: During model training During the process, record whether each batch includes data of a specific category, as well as the parameter update value corresponding to each batch. , Indicates in Rounds and The parameter update amount in the batch. If the data of a specific category is not included, this step is skipped; (2) Determine the batches of data of a specific category: Find all batches that include data of a specific category from the records , Is a collection that includes all batch indexes of a specific category of data; (3) Undo parameter updates related to a specific category: Undo the current parameters of the model Subtract the parameter update value related to the specific category from the original value, that is: (2); In formula (2), are the parameters of the model after training. are the model parameters after the forgetting operation, is the sum of parameter updates for all batches that include data of a particular class.

[0011] Furthermore, the KL divergence forgetting method specifically includes: (11) Using the initial model Predict the output distribution of a specific target class And the output distribution of the corresponding sample category in another dataset ; (22) Calculate KL divergence to measure the two probability distributions and The differences between: (3); In formula (3), It is the predicted probability of the i-th category output by the initial model M when the input is a target specific category sample; It is the predicted probability of the i-th category output by the initial model M for the sample category with the same semantics as the target category; (33) Adjust the model parameters by minimizing the KL divergence, forcing the model to target the output distribution of specific categories Distribution to QMNIST Alignment, thereby eliminating the unique features of the target class and achieving the effect of forgetting the target class; (44) By simply fine-tuning the model, the recognition ability of other categories can be restored: Excluding the training data for the forgotten classes, a small amount of training data is extracted from the remaining original training data, and a learning rate threshold and number of iterations are set. Fine-tuning is performed using the same optimizer as the original training. During fine-tuning, the model calculates the loss function through forward propagation and updates the parameters through backpropagation, ensuring that the model forgets specific data while restoring its ability to recognize other classes. Initial Model The forgetting model is obtained by the above two forgetting methods ,at this time It has the ability to achieve a recognition accuracy of close to 0 for category 2 and a high recognition accuracy for other categories.

[0012] Furthermore, the S3 specifically includes: S31, selecting the remaining category data from the original data set as incremental learning data and adding a certain amount of target forgotten category data as input data for incremental learning; S32. Using the forgetting model As the teacher model, initialize the student model , so that it inherits the convolutional layer parameters of the teacher model, selectively initializes the fully connected layer parameters, and then performs knowledge distillation training; S33. In each training batch, for the new data x continuously learned in the incremental learning process, the output of the teacher model is calculated. The output logits of the teacher model are defined as : (4); In each training batch, for the new data x continuously learned during the incremental learning process, the output of the student model is calculated. The output logits of the student model are defined as : (5); S34. Calculate the knowledge distillation loss, guide the student model to keep the forgetting state through the soft label of the teacher model, and the knowledge distillation loss Defined as: (6); In formula (6), T is the temperature parameter used to smooth the output distribution. The role of the temperature parameter T is to make the output distribution of the teacher model smoother, making it easier for the student model to learn.

[0013] S35. Calculate the total loss function, combining cross entropy loss and knowledge distillation loss: (7); In formula (7), α is a hyperparameter used to balance the weights of the two losses.

[0014] S36, total loss of use Perform backpropagation and update the parameters of the student model: (8); In formula (8), η is the learning rate, Represents the set of trainable parameters of the student model. In the incremental learning phase, the parameters are gradually updated through optimization. Represents the loss function for the student model parameters gradient.

[0015] S37. Self-set iterative rounds. After each training round, verify the overall loss and accuracy of the model on the test set. If the test accuracy of the current round is higher than the best accuracy recorded previously, update the best model state. After all training rounds are completed, save the model state with the highest test accuracy as the final model.

[0016] In another aspect of the present invention, a device for solving the method of forgetting learning catastrophic recall based on knowledge distillation is provided, the device comprising: at least one processor; and A memory storing instructions, which, when executed by the at least one processor, causes the at least one processor to execute the method for solving catastrophic recall of forgotten learning based on knowledge distillation as described above.

[0017] In another aspect of the present invention, a computer-readable storage medium is provided, which stores executable instructions. When the instructions are executed, the machine executes the method for solving catastrophic recall of forgotten learning based on knowledge distillation as described above.

[0018] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention provides a method for solving catastrophic recall in forgetting learning based on knowledge distillation. It adopts two different forgetting learning methods, "amnesia forgetting learning" and "KL divergence fitting method". After the forgetting request, the accuracy of the target category will drop rapidly and tend to zero. Amnesia forgetting learning effectively eliminates the influence of the target category data in the model.

[0019] (2) The present invention provides a method for solving catastrophic recall in forgotten learning based on knowledge distillation. By adding knowledge distillation to the incremental learning process, the model is suppressed by the features of the distillation model during the incremental process, so that the model maintains a low accuracy rate in the forgotten category after incremental learning, while the accuracy rate of other categories remains unchanged. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of feature suppression based on knowledge distillation proposed in the present invention; Figure 2 This is the overall framework diagram of feature suppression based on knowledge distillation proposed in this invention; Figure 3 This is the experimental result of the classification accuracy of the initial model in Example 1 of the present invention as the number of training rounds on the test set changes; Figure 4 This is the experimental result of the change in the accuracy of the model after applying the "amnesia forgetting learning" method in Example 1 of the present invention; Figure 5 This is the experimental result of the change in model accuracy after applying the KL divergence forgetting method in Example 1 of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0023] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0024] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0025] Example 1 Ginseng Figure 1 This embodiment provides a method for solving catastrophic recall caused by forgetting learning based on knowledge distillation, the method comprising: Step S1, initial model and dataset selection and full-category high-precision recognition model generation stage: select the initial model and use the standard dataset for training, optimize the initial model parameters by minimizing the cross-entropy loss, and generate a full-category high-precision recognition model.

[0026] Preferably, the S1 specifically includes: S11. Select a standard dataset, i.e., an original dataset, such as the MNIST dataset or the CIFAR-10 dataset, and use it as training data. S12. Choose a classic convolutional neural network architecture (such as LeNet-5) as the initial model ; S13. Train the initial model using each category of the dataset , so that its recognition ability for each category can reach high accuracy, including; category one, category two, category three. The training process optimizes the model parameters by minimizing the cross entropy loss LCE: (1); In formula (1), is the input data, is the label corresponding to the input data (one-hot encoding), is the total number of categories, Is the initial model For input data Belong to category The predicted probability of .

[0027] Step S2, target category image data input - parameter backtracking / KL divergence distribution alignment - forgetting category strong suppression model generation stage: select the target forgotten category, apply the forgetting method to adjust the model parameters, and generate a forgetting model whose recognition ability for the target forgotten category is lower than the recognition accuracy for other categories; preferably, the generated forgetting model has a recognition ability for the target forgotten category close to 0, while the recognition accuracy for other categories is extremely high, that is, a forgetting category strong suppression model, where extremely high generally means an accuracy rate higher than 90%.

[0028] Preferably, the S2 specifically includes: S21. Select a target specific category from the data set as the forgotten target category. For example, category 2 is selected as the forgotten target category, and the operation of forgetting this category is subsequently performed.

[0029] S22. Simultaneously apply two forgetting methods to forget: one is amnesia forgetting learning, and the other is KL divergence forgetting method. The present invention proposes a double forgetting method for this task.

[0030] The "amnesiac forgetting learning" method records each small batch of samples and their corresponding parameter updates during training. When a request is received to forget category 2, the parameter updates of the batch that includes category 2 samples are backtracked to remove the learning traces of the specific data, namely category 2. The specific steps of this method are as follows: (1) Record the training process: During model training During the process, record whether each batch includes category 2 data, as well as the parameter update values ​​corresponding to each batch. Here Indicates in Rounds and The number of parameter updates in a batch.

[0031] (2) Determine the batches of Category 2 data: Find all batches that include Category 2 data from the records. , It is a collection that includes the batch indexes of all category 2 data.

[0032] (3) Undo the parameter updates related to category 2: From the current parameters of the model Subtract the parameter update value related to category 2 from the original value, that is: (2); In formula (2), are the parameters of the model after training. are the model parameters after the forgetting operation, is the sum of parameter updates for all batches that include class 2 data.

[0033] KL divergence forgetting method: select another dataset, such as the QMNIST dataset, and select a sample category from this QMNIST dataset whose similarity with the forgotten target category is greater than the set threshold. The number of sample categories is the same as the forgotten target category. Through fitting, that is, KL divergence, the sample category is aligned with the output distribution of the target category, which further reduces the model's recognition ability for category 2.

[0034] The specific steps of the KL divergence forgetting method are as follows: (1) Using the initial model Predict the output distribution of target category two And the output distribution of the corresponding sample category in another dataset, the QMNIST dataset .

[0035] (2) Calculate KL divergence to measure the two probability distributions and The differences between: (3); In formula (3), It is the predicted probability of the i-th category output by the initial model M when the input is a target specific category sample; It is the predicted probability of the i-th category output by the initial model M for the sample category with the same semantics as the target category; (3) Adjust the model parameters by minimizing the KL divergence, forcing the model to distribute the output of target category 2 Distribution to QMNIST Alignment eliminates the unique features of the target class and achieves the effect of forgetting the target class.

[0036] (4) Restore the recognition ability of other categories by simply fine-tuning the model. Exclude the training data of the forgotten class, extract a small amount of training data from the remaining original training data, set a smaller learning rate and fewer iterations, and use the same optimizer as the original training for fine-tuning. At the same time, during the fine-tuning process, the model calculates the loss function through forward propagation and updates the parameters through backpropagation to ensure that the model restores the recognition ability of other categories while forgetting specific data.

[0037] Initial Model The forgetting model is obtained by the above two forgetting methods ,like Figure 2 As shown, at this time It has the ability to achieve a recognition accuracy of close to 0 for category 2 and a high recognition accuracy for other categories.

[0038] Step S3, input data confirmation - Logits distribution alignment based on knowledge distillation constraints - generation of catastrophic recall suppression incremental model generation phase: The remaining category data except the target forgotten category is selected from the original dataset as incremental learning data and a certain amount (e.g., 10%) of category 2 data is added as input data to simulate the actual scenario. The generated forgetting model is used as the teacher model to initialize the student model. The student model constrains the output distribution through the knowledge distillation loss on the input data and updates the parameters of the student model.

[0039] Furthermore, if Figure 2 As shown in the incremental learning stage, S3 specifically includes: S31, selecting the remaining category data from the original data set as incremental learning data and adding a certain amount of target forgotten category data as input data for incremental learning; S32. Using the forgetting model As the teacher model, initialize the student model , so that it inherits the convolutional layer parameters of the teacher model, selectively initializes the fully connected layer parameters, and then performs knowledge distillation training; S33. In each training batch, for the new data x continuously learned in the incremental learning process, the output of the teacher model is calculated. The output logits of the teacher model are defined as : (4); In each training batch, for the new data x continuously learned during the incremental learning process, the output of the student model is calculated. The output logits of the student model are defined as : (5); S34. Calculate the knowledge distillation loss, guide the student model to keep the forgetting state through the soft label of the teacher model, and the knowledge distillation loss Defined as: (6); In formula (6), T is the temperature parameter used to smooth the output distribution. The role of the temperature parameter T is to make the output distribution of the teacher model smoother, making it easier for the student model to learn.

[0040] S35. Calculate the total loss function, combining cross entropy loss and knowledge distillation loss: (7); In formula (7), α is a hyperparameter used to balance the weights of the two losses.

[0041] S36, total loss of use Perform backpropagation and update the parameters of the student model: (8); In formula (8), η is the learning rate, Represents the set of trainable parameters of the student model, which are gradually updated through optimization during the incremental learning phase. Represents the loss function for the student model parameters gradient.

[0042] S37. Self-set iterative rounds. After each training round, verify the overall loss and accuracy of the model on the test set. At the same time, extract the test accuracy and loss of the target forgotten category. If the test accuracy and loss of the target forgotten category in the current round are better than the best accuracy and loss recorded before, update the best model state. After all training rounds, save the student model state with the highest test accuracy as the final model.

[0043] At this point, we have obtained an incremental model, namely the final model, which can both learn new knowledge and suppress forgotten categories to ensure the persistence of the forgetting effect.

[0044] Furthermore, this embodiment applies this method to medical data security, trains an initial diagnostic model on a full-category medical dataset, performs target category suppression on the patient data that is specified to be forgotten to generate a forgetting model, accesses incremental medical data and a small amount of deleted patient data (which inevitably leaves behind some data and data that is extremely similar to the forgotten data), and generates a catastrophic recall immune diagnostic model through knowledge distillation constrained Logits distribution alignment.

[0045] The experimental process is as follows: This paper conducts experiments on the MNIST dataset, a 10-category dataset of handwritten images consisting of 60,000 training images and 10,000 test images. These images are grayscale, each with a resolution of 1x28x28 pixels. The experiments are performed on the Lenet-5 convolutional neural network, a classic neural network learning architecture primarily used for handwritten digit recognition tasks.

[0046] Since the process of the present invention is a series of tasks from training the initial model, forgetting the data points to obtain the forgotten model, and then incremental learning to obtain the new model. Therefore, the present invention will gradually present its results. All experiments of the present invention are to remove category 2 from the data set. First, the present invention selects to learn 50% of the data in all categories to obtain the initial model. Then, the present invention uses two forgetting methods to forget category 2. In the subsequent incremental learning, the present invention learns the remaining 50% of the data in other categories and extracts a small amount of 10% of the data from category 2 for learning. Then, the present invention observes the changes in the model.

[0047] like Figure 3 The figure shows how classification accuracy changes with the number of training rounds on the test set when using the Lenet-5 model trained on the MNIST dataset. The horizontal axis represents the number of training rounds, which totals 10 rounds, while the vertical axis represents the accuracy (%), ranging from 0% to 100%. As can be seen from the figure, the accuracy of the model on the test set gradually improves with the increase in training rounds. Initially (two training rounds), the accuracy is approximately 92%. As the number of training rounds increases, the accuracy steadily increases, ultimately approaching 100% after 10 training rounds. This demonstrates that the Lenet-5 model performs well on the MNIST dataset and is capable of effectively learning and improving classification performance.

[0048] Then, the present invention adopts two different forgetting learning methods, one is "amnesia forgetting learning" and the other is KL divergence fitting method. Figure 4 As shown in the figure, after applying the "amnesiac forgetting learning" method, the model's accuracy changes across different training rounds. The two curves in the figure represent the target category accuracy, that is, the model's recognition accuracy for the data category to be forgotten or deleted; and the other category accuracy, that is, the model's recognition accuracy for non-target category data. Before the data removal request, the accuracy of both the target category and the other category is relatively high, with little fluctuation. After the forget request, the target category accuracy drops rapidly and approaches zero, indicating that amnesiac forgetting learning effectively eliminates the influence of the target category data in the model.

[0049] like Figure 5 As shown in the figure, after applying the KL divergence forgetting method, the accuracy of both the target class and other classes was relatively high before the data removal request. After the data removal request, the accuracy of the target class rapidly declined due to the influence of KL divergence, while the accuracy of other classes was also affected to some extent by the changes in model parameters. After three rounds of adjusting the KL divergence of the target class and two rounds of simple data fine-tuning, the model accuracy of other classes recovered, and the target class remained stable in the forgotten state.

[0050] After incremental learning with the remaining 50% of the data from other categories and a small amount of 10% from category 2, the results are shown in Tables 1 and 2 (for the normal incremental learning section). The confusion matrix evaluates the model's performance on the dataset and shows how the model's predictions for each category compare to the actual labels. The rows of the matrix represent the actual categories, the columns represent the predicted categories, and the number in each cell indicates the number of samples that were predicted to be in the actual category. As can be seen from these two figures, the accuracy of the forgotten model for the forgotten category 2 is already very high after incremental learning. This indicates that due to the mixing of the small amount of removed data, the performance of the forgotten model for the forgotten category improves rapidly after incremental learning.

[0051] By incorporating knowledge distillation into the incremental learning process, the incremental learning results based on knowledge distillation are very significant, as shown in Tables 1 and 2. Although the forgotten model incorporates a small amount of data from the forgotten categories during incremental learning, the model is suppressed by the characteristics of the distillation model during the incremental learning process. This allows the model to maintain a low accuracy rate (close to 0%) on the forgotten categories after incremental learning, while maintaining the same accuracy rate for other categories. This demonstrates the effectiveness of our method.

[0052] Table 1 Experimental results of the change of accuracy of category 2 with training rounds through normal incremental learning of the forgetting model after "amnesia" and incremental learning of the present invention

[0053] Table 2 Experimental results of the normal incremental learning of the forgetting model after KL divergence forgetting and the incremental learning of the present invention for the accuracy of category 2 as the training rounds change

[0054] Example 2 This embodiment provides a device for implementing a method for solving catastrophic recall in forgotten learning based on knowledge distillation, the device comprising: at least one processor; and, A memory storing instructions, which, when executed by the at least one processor, causes the at least one processor to perform the method for solving catastrophic recall of forgotten learning based on knowledge distillation as described above.

[0055] In this embodiment, electronic devices include but are not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, etc.

[0056] Example 3 This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed, enable the machine to perform the method for solving catastrophic recall of forgotten learning based on knowledge distillation as described above.

[0057] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

[0058] In this case, the program code itself read from the computer-readable medium can realize the function of any one of the above embodiments, and thus the computer-readable code and the computer-readable storage medium storing the computer-readable code constitute part of this specification.

[0059] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0060] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A method based on knowledge distillation to solve the problem of catastrophic recall in forgotten learning, characterized by: The method comprises: S1. Select an initial model and train it using a standard dataset, optimizing the initial model parameters by minimizing the cross entropy loss. S2. Select the target forgetting category and adjust the model parameters using the forgetting method to generate a forgetting model whose recognition ability for the target forgetting category is lower than that for other categories; S3. Select the remaining category data except the target forgotten category from the original dataset as incremental learning data and add a certain amount of target forgotten category data as input data. Use the generated forgetting model as the teacher model to initialize the student model. The student model constrains the output distribution on the input data through knowledge distillation loss and updates the parameters of the student model.

2. The method for solving catastrophic recall in forgotten learning based on knowledge distillation according to claim 1 is characterized in that: Said S1 specifically includes: S11. Select a standard image dataset as training data; S12. Select a convolutional neural network architecture as the initial model ; S13. Train the initial model using each category of the dataset , so that its recognition ability for each category is improved. The training process optimizes the initial model parameters by minimizing the cross entropy loss LCE: （1）； In formula (1), is the input data, is the true label corresponding to the input data, The true label Components or categories The label value of is the total number of categories, Is the initial model For input data Belong to category The predicted probability of .

3. The method for solving catastrophic recall in forgotten learning based on knowledge distillation according to claim 1 or 2, characterized in that: The S2 specifically includes: S21, select a target specific category from the data set as the forgotten target category, and then perform a forgetting operation on this category; S22. Simultaneously apply two forgetting methods: one is amnesia forgetting learning, and the other is KL divergence forgetting method; Amnesiac forgetting learning method: Record each mini-batch of samples and their corresponding parameter updates during training. When receiving a request to forget a specific category, backtrack the parameter updates of the batch that includes samples of the specific category, thereby removing the learning traces of the data of the forgotten target category. KL divergence forgetting method: select another data set, and select a sample category from this data set whose similarity with the forgotten target category is greater than the set threshold, and the number of sample categories is the same as the forgotten target category. Through fitting, that is, KL divergence, the sample category is aligned with the output distribution of the target category, so that the model's recognition ability of the forgotten target category is further reduced.

4. The method for solving catastrophic recall in forgotten learning based on knowledge distillation according to claim 3 is characterized in that: The amnesia forgetting learning specifically includes: (1) Record the training process: During model training During the process, record whether each batch includes data of a specific category, as well as the parameter update value corresponding to each batch. , Indicates in Rounds and The parameter update amount in the batch. If the data of a specific category is not included, this step is skipped; (2) Determine the batches of data of a specific category: Find all batches that include data of a specific category from the records , Is a collection that includes all batch indexes of a specific category of data; (3) Undo parameter updates related to a specific category: Undo the current parameters of the model Subtract the parameter update value related to the specific category from the original value, that is: （2）； In formula (2), are the parameters of the model after training. are the model parameters after the forgetting operation, is the sum of parameter updates for all batches that include data of a particular class.

5. The method for solving catastrophic recall in forgotten learning based on knowledge distillation according to claim 4 is characterized in that: The KL divergence forgetting method specifically includes: (11) Using the initial model Predict the output distribution of a specific target class And the output distribution of the corresponding sample category in another dataset ; (22) Calculate KL divergence to measure the two probability distributions and The differences between: （3）； In formula (3), It is the predicted probability of the i-th category output by the initial model M when the input is a target specific category sample; It is the predicted probability of the i-th category output by the initial model M for the sample category with the same semantics as the target category; (33) Adjust the model parameters by minimizing the KL divergence, forcing the model to target the output distribution of specific categories Distribution to QMNIST Alignment, eliminating unique features that mark specific categories; (44) Excluding the training data of the forgotten class, extracting a small amount of training data from the remaining original training data and setting a learning rate threshold and iteration rounds, fine-tuning is performed using the same optimizer as the original training; at the same time, during the fine-tuning process, the model calculates the loss function through forward propagation and updates the parameters through backpropagation; Initial Model The forgetting model is obtained by the above two forgetting methods .

6. The method for solving catastrophic recall in forgotten learning based on knowledge distillation according to claim 5, characterized in that: The S3 specifically includes: S31, selecting the remaining category data from the original data set as incremental learning data and adding a certain amount of target forgotten category data as input data for incremental learning; S32. Using the forgetting model As the teacher model, initialize the student model , let it inherit the convolutional layer parameters of the teacher model, selectively initialize the fully connected layer parameters, and then perform knowledge distillation training; S33. In each training batch, for the new data x continuously learned in the incremental learning process, the output of the teacher model is calculated. The output logits of the teacher model are defined as : （4）； In each training batch, for the new data x continuously learned during the incremental learning process, the output of the student model is calculated. The output logits of the student model are defined as : （5）； S34. Calculate the knowledge distillation loss, guide the student model to keep the forgetting state through the soft label of the teacher model, and the knowledge distillation loss Defined as: （6）； In formula (6), T is the temperature parameter used to smooth the output distribution. The role of the temperature parameter T is to make the output distribution of the teacher model smoother, so that it is easier to be learned by the student model; S35. Calculate the total loss function, combining cross entropy loss and knowledge distillation loss: （7）； In formula (7), α is a hyperparameter used to balance the weights of the two losses; S36, total loss of use Perform backpropagation and update the parameters of the student model: （8）； In formula (8), η is the learning rate, Represents the set of trainable parameters of the student model. In the incremental learning phase, the parameters are gradually updated through optimization. Represents the loss function for the student model parameters gradient; S37. Self-set iterative rounds. After each training round, verify the overall loss and accuracy of the model on the test set. At the same time, extract the test accuracy and loss of the target forgotten category. If the test accuracy and loss of the target forgotten category in the current round are better than the best accuracy and loss recorded before, update the best model state. After all training rounds, save the student model state with the highest test accuracy as the final model.

7. The method for solving catastrophic recall in forgotten learning based on knowledge distillation according to claim 2, characterized in that: The labels are one-hot encoded.

8. A device for implementing a method for solving catastrophic recall of forgotten learning based on knowledge distillation, characterized in that: The device comprises: processor; a memory having stored thereon a computer program executable on the processor; Wherein, when the computer program is executed by the processor, the steps of the method for solving catastrophic recall of forgotten learning based on knowledge distillation as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images