A method and apparatus for resolving forgetting and catastrophic memory based on knowledge distillation.

Abstract

This invention belongs to the technical field of information and data processing, and more specifically, relates to a method and apparatus for solving catastrophic recall in forgetting learning based on knowledge distillation. The method includes selecting an initial model and training it using a standard dataset, optimizing the initial model parameters by minimizing cross-entropy loss; selecting a target forgetting category, applying a forgetting-like method to adjust the model parameters, generating a forgetting model with low recognition ability for the target forgetting category but high accuracy in recognizing other categories; selecting the remaining category data from the original dataset as incremental learning data and adding a quantitative amount of target forgetting category data as input data; using the generated forgetting model as the teacher model to initialize the student model; the student model constrains the output distribution on the input data through knowledge distillation loss, and updates the parameters of the student model. This invention solves the problem that forgotten models cannot maintain forgetting performance during incremental learning, and may experience catastrophic forgetting when absorbing new information.

Description

Technical Field

[0001] This invention belongs to the technical field of information and data processing, and more specifically, relates to a method and apparatus for solving catastrophic recall problems in learning based on knowledge distillation. Background Technology

[0002] The rapid development of artificial intelligence and machine learning technologies has greatly boosted social productivity. However, the widespread application of these technologies has also raised significant concerns about data privacy and security. Machine learning models typically rely on large amounts of training data to improve performance, but this data may contain sensitive user information. This necessitates that machine learning models possess the ability to remove specific data knowledge without retraining, thus giving rise to research into "machine forgetting" technology.

[0003] Many approximate forgetting methods have emerged in recent years. The PUMA method achieves approximate forgetting by simulating the impact of each training data point on model performance. "Amnesiac forgetting learning" methods prompt the model during training to record which batches contain sensitive data and corresponding parameter updates; if a data removal request occurs, parameter updates will only apply to the specified batches. However, existing research on forgetting methods focuses only on a single forgetting task, neglecting consideration of subsequent incremental learning. Regarding data security, Wang et al. focused on the problem of data remnants, i.e., the possibility that the model may retain these traces despite efforts to eliminate them. Liu et al. proposed a unified workflow involving machine forgetting, comprising three stages: a training stage, a forgetting learning stage, and a post-forgetting learning stage. The post-forgetting learning stage requires the model to provide inference services, and the subsequent behavior of the forgotten model has attracted increasing attention.

[0004] Chinese invention patent CN118298279A discloses a method, system, medium, and device for knowledge distillation of cross-domain passive domain data. The method includes: using a pre-trained teacher model to first train an image generator, then an image encoder, while simultaneously warming up a student model, and updating parameters separately using their respective loss functions; training a masking network on a student domain training dataset using the image generator, image encoder, and teacher model, and then training a student model on the same dataset, adjusting the weights of domain-specific information loss and cross-domain knowledge distillation loss through a stage factor; and inputting the image to be predicted into the trained student model to obtain the classification label of the image.

[0005] This shows that some research progress has been made in this field in recent years, mainly focusing on improving the efficiency of forgetting algorithms and the effectiveness of data removal. However, the problem that forgotten models cannot maintain their forgetting performance during incremental learning has been overlooked. Incremental learning refers to the process where a model learns new knowledge by gradually learning new tasks or categories without forgetting old knowledge. However, existing research has not fully explored the performance of forgotten models after incremental learning. In fact, models may experience catastrophic recall problems when absorbing new information. Summary of the Invention

[0006] The present invention aims to overcome at least one of the defects of the prior art and provide a method for solving catastrophic recall in forgetting learning based on knowledge distillation, in order to solve the problem of catastrophic recall, that is, the model may unintentionally relearn the features of forgotten categories during the incremental learning process, thereby leading to forgetting failure.

[0007] The detailed technical solution of this invention is as follows:

[0008] A method for addressing catastrophic recall in learning based on knowledge distillation, the method comprising:

[0009] S1. Select an initial model and train it using a standard dataset, optimizing the initial model parameters by minimizing the cross-entropy loss;

[0010] S2. Select the target forgetting category, apply the forgetting-like method to adjust the model parameters, and generate a forgetting model whose ability to identify the target forgetting category is lower than its accuracy in identifying other categories.

[0011] S3. Select the remaining category data (excluding the target forgetting category) from the original dataset as incremental learning data and add a quantitative amount of target forgetting category data as input data. Use the generated forgetting model as the teacher model to initialize the student model. The student model constrains the output distribution on the input data through knowledge distillation loss and updates the parameters of the student model.

[0012] Furthermore, S1 specifically includes:

[0013] S11. Select a standard image dataset and use it as training data;

[0014] S12. Select a convolutional neural network architecture as the initial model. ;

[0015] S13. Train the initial model using each category of the dataset. This enhances its ability to identify each category. The training process optimizes the initial model parameters by minimizing the cross-entropy loss (LCE).

[0016] (1);

[0017] In formula (1), It is the input data. These are the labels corresponding to the input data. It is the total number of categories. It is the initial model For input data Category The predicted probability. Preferably, the label is one-hot encoded.

[0018] Furthermore, S2 specifically includes:

[0019] S21. Select a specific category from the dataset as the target category for forgetting, and then perform the forgetting operation on this category.

[0020] S22. Simultaneously apply two forgetting methods: one is amnesia-based forgetting learning, and the other is the KL divergence forgetting method;

[0021] Amnesia-based learning method: Record each mini-batch of samples and its corresponding parameter updates during training. When a request to forget a specific category is received, backtrack the parameter updates of the batch including samples of the specific category, thereby removing the learning traces of the data of the forgotten target category.

[0022] The KL divergence forgetting method involves selecting another dataset and choosing a sample category from this dataset whose similarity to the forgotten target category is greater than a set threshold. The number of sample categories is the same as the number of forgotten target categories. By fitting the KL divergence, the sample categories are aligned with the output distribution of the target category, further reducing the model's ability to identify the forgotten target category.

[0023] Furthermore, the aforementioned amnesia-induced forgetting learning specifically includes:

[0024] (1) Record the training process: during model training During the process, it records whether each batch includes data of a specific category, and the corresponding parameter update values ​​for each batch. , Indicates the first The round and the If the parameter update amount in a batch does not include data of a specific category, skip this step.

[0025] (2) Identify batches of data in a specific category: Find all batches of data in a specific category from the records. , It is a collection that includes batch indexes of all data of a specific category;

[0026] (3) Undo parameter updates related to a specific category: from the model's current parameters Subtract the updated values ​​of parameters related to a specific category from the middle, that is:

[0027] (2);

[0028] In formula (2), These are the parameters of the model after training. These are the model parameters after the forgetting operation. It is the sum of parameter update values ​​for all batches that include data of a specific category.

[0029] Furthermore, the KL divergence forgetting method specifically includes:

[0030] (11) Using the initial model Predict the output distribution of a specific target category and the output distribution of the corresponding sample categories in another dataset. ;

[0031] (22) Calculate the KL divergence to measure the probability distributions of two probability distributions. and Differences between them:

[0032] (3);

[0033] In formula (3), It is the predicted probability of the i-th class output by the initial model M when the input is a sample of a specific class of the target; It is the predicted probability of the i-th class output by the initial model M for sample classes with the same semantics as the target class;

[0034] (33) By minimizing the KL divergence, the model parameters are adjusted to force the model to distribute the output of the target specific class. Distribution to QMNIST Alignment eliminates the unique characteristics of the target class, achieving the effect of forgetting the target class;

[0035] (44) By simply fine-tuning the model, the ability to recognize other categories can be restored:

[0036] Excluding the training data of the forgotten classes, a small amount of training data is extracted from the remaining original training data, and a learning rate threshold and iteration epochs are set. Fine-tuning is performed using the same optimizer as the original training. At the same time, during the fine-tuning process, the model calculates the loss function through forward propagation and updates the parameters through backpropagation to ensure that the model can recover its ability to recognize other classes while forgetting specific data.

[0037] Initial model The forgetting model was obtained through the two forgetting methods described above. ,at this time It has the ability to achieve near-zero accuracy in identifying Category 2, while maintaining high accuracy in identifying other categories.

[0038] Furthermore, S3 specifically includes:

[0039] S31. Select the remaining category data from the original dataset as incremental learning data and add a quantitative amount of target forgotten category data as input data for incremental learning;

[0040] S32. Using the forgetting model As a teacher model, initialize the student model. This allows it to inherit the convolutional layer parameters of the teacher model, selectively initialize the fully connected layer parameters, and then perform knowledge distillation training.

[0041] S33. In each training batch, for the new data x learned during the incremental learning process, calculate the output of the teacher model. The output logits of the teacher model is defined as... :

[0042] (4);

[0043] In each training batch, for each new data x learned during incremental learning, the output of the student model is calculated. The output logits of the student model is defined as follows: :

[0044] (5);

[0045] S34. Calculate the knowledge distillation loss. The teacher model uses soft labels to guide the student model in maintaining a forgotten state. (Knowledge distillation loss is then calculated.) Defined as:

[0046] (6);

[0047] In formula (6), T is a temperature parameter used to smooth the output distribution. The role of the temperature parameter T is to make the output distribution of the teacher model smoother, thus making it easier for the student model to learn.

[0048] S35. Calculate the total loss function, combining cross-entropy loss and knowledge distillation loss:

[0049] (7);

[0050] In formula (7), α is a hyperparameter used to balance the weights of the two losses.

[0051] S36, Total Losses Due to Use Perform backpropagation and update the parameters of the student model:

[0052] (8);

[0053] In formula (8), η is the learning rate. This represents the set of trainable parameters for the student model, which are gradually updated through optimization during the incremental learning phase. This indicates that the loss function applies to the student model parameters. The gradient.

[0054] S37. The iterative rounds are set by the user. After each training round, the overall loss and accuracy of the model on the test set are verified. If the test accuracy of the current round is higher than the previously recorded best accuracy, the best model state is updated. After all training rounds are completed, the model state with the highest test accuracy is saved as the final model.

[0055] In another aspect of the invention, an apparatus is provided for a method of resolving catastrophic recall in forgetting learning based on knowledge distillation, the apparatus comprising:

[0056] At least one processor; and

[0057] A memory that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform a method for resolving catastrophic recall of forgotten learning based on knowledge distillation, as described above.

[0058] In another aspect of the invention, a computer-readable storage medium is also provided, which stores executable instructions that, when executed, cause the machine to perform a method for resolving catastrophic recall of forgotten learning based on knowledge distillation as described above.

[0059] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0060] (1) The present invention provides a method for solving the catastrophic memory problem of forgetting learning based on knowledge distillation. It adopts two different forgetting learning methods, "amnesia-based forgetting learning" and "KL divergence fitting method". After the forgetting request, the accuracy of the target category will drop rapidly and tend to zero. Amnesia-based forgetting learning effectively eliminates the influence of the target category data in the model.

[0061] (2) The present invention provides a method for solving catastrophic recall of forgotten learning based on knowledge distillation. By adding knowledge distillation during the incremental learning process, the model receives feature suppression from the distillation model during the incremental process, so that the model still maintains a low accuracy in the forgotten category after incremental learning, while the accuracy in other categories remains unchanged. Attached Figure Description

[0062] Figure 1 is the flow chart of the feature suppression based on knowledge distillation proposed by the present invention;

[0063] Figure 2 is the overall framework diagram of the feature suppression based on knowledge distillation proposed by the present invention;

[0064] Figure 3 is the experimental result of the change of the classification accuracy of the initial model in Embodiment 1 of the present invention with the number of training rounds on the test set;

[0065] Figure 4 is the experimental result of the change of the accuracy of the model after applying the "amnesic forgetting learning" method in Embodiment 1 of the present invention;

[0066] Figure 5 is the experimental result of the change of the accuracy of the model after applying the KL divergence forgetting method in Embodiment 1 of the present invention. Detailed implementation manners

[0067] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0068] It should be noted that the following detailed descriptions are all exemplary and intended to provide further explanations of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0069] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0070] Without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0071] Embodiment 1

[0072] Refer Figure 1 , this embodiment provides a method for solving catastrophic forgetting in forgetting learning based on knowledge distillation, and the method includes:

[0073] Step S1, the initial model and dataset selection and the generation stage of the all-class high-precision recognition model: Select an initial model and train it using a standard dataset, and optimize the parameters of the initial model by minimizing the cross-entropy loss to generate an all-class high-precision recognition model.

[0074] Preferably, S1 specifically includes:

[0075] S11. Select a standard dataset, i.e., the original dataset, such as the MNIST dataset or the CIFAR-10 dataset, and use it as the training data.

[0076] S12. Choose a classic convolutional neural network architecture (such as LeNet-5) as the initial model. ;

[0077] S13. Train the initial model using each category of the dataset. This enables it to achieve high accuracy in recognizing each category, including Category 1, Category 2, and Category 3. The training process optimizes the model parameters by minimizing the cross-entropy loss (LCE).

[0078] (1);

[0079] In formula (1), It is the input data. It is the label (one-hot encoded) corresponding to the input data. It is the total number of categories. It is the initial model For input data Category The predicted probability.

[0080] Step S2, Target Category Image Data Input - Parameter Backtracking / KL Divergence Distribution Alignment - Strong Suppression Model Generation Stage for Forgotten Category: Select the target forgotten category, apply a forgetting-like method to adjust the model parameters, and generate a forgetting model whose ability to identify the target forgotten category is lower than its accuracy in identifying other categories; preferably, the generated forgetting model whose ability to identify the target forgotten category is close to 0, while its accuracy in identifying other categories is extremely high, which is the strong suppression model for forgotten category. Here, extremely high generally means an accuracy higher than 90%.

[0081] Preferably, S2 specifically includes:

[0082] S21. Select a specific category from the dataset as the target category for forgetting. For example, select category 2 as the target category for forgetting, and then perform the forgetting operation on this category.

[0083] S22. Simultaneously apply two forgetting methods for forgetting: one is amnesia-based forgetting learning, and the other is the KL divergence forgetting method. This invention proposes a dual forgetting method for this task.

[0084] The "amnesia-based learning" method records each mini-batch of samples and its corresponding parameter updates during training. Upon receiving a request to forget category two data, it backtracks the parameter updates of the batch including category two samples, thereby removing the learning traces of that specific data, i.e., category two. The specific steps of this method are as follows:

[0085] (1) Record the training process: during model training During the process, record whether each batch includes data from category two, and the corresponding parameter update values ​​for each batch. Here Indicates the first The round and the The number of parameter updates in each batch.

[0086] (2) Identify the batches of Category 2 data: Find all batches that include Category 2 data from the records. , It is a collection that includes batch indexes of all Category 2 data.

[0087] (3) Cancel parameter updates related to Category 2: from the current parameters of the model Subtract the updated parameter values ​​related to category two from the middle, that is:

[0088] (2);

[0089] In formula (2), These are the parameters of the model after training. These are the model parameters after the forgetting operation. It is the sum of parameter update values ​​for all batches including Category 2 data.

[0090] KL divergence forgetting method: Select another dataset, such as the QMNIST dataset, and select a sample class from the QMNIST dataset whose similarity to the forgotten target class is greater than a set threshold. The number of sample classes is the same as the number of forgotten target classes. By fitting the KL divergence, the sample classes are aligned with the output distribution of the target class, which further reduces the model's ability to recognize the second category.

[0091] The specific steps of the KL divergence forgetting method are as follows:

[0092] (1) Using the initial model Predict the output distribution of target category 2 And the output distribution of the corresponding sample categories in another dataset, namely the QMNIST dataset. .

[0093] (2) Calculate the KL divergence to measure the probability distributions of two probability distributions. and Differences between them:

[0094] (3);

[0095] In formula (3), It is the predicted probability of the i-th class output by the initial model M when the input is a sample of a specific class of the target; It is the predicted probability of the i-th class output by the initial model M for sample classes with the same semantics as the target class;

[0096] (3) By minimizing the KL divergence, the model parameters are adjusted to force the model to distribute the output of the target class II. Distribution to QMNIST Alignment eliminates the unique characteristics of the target class, achieving the effect of forgetting the target class.

[0097] (4) The model is fine-tuned to restore its ability to recognize other categories. The training data of the forgotten class is excluded. A small amount of training data is extracted from the remaining original training data and a small learning rate and fewer iterations are set. The same optimizer as the original training is used for fine-tuning. At the same time, during the fine-tuning process, the model calculates the loss function through forward propagation and updates the parameters through backpropagation to ensure that the model can restore its ability to recognize other categories while forgetting specific data.

[0098] Initial model The forgetting model was obtained through the two forgetting methods described above. ,like Figure 2 As shown, at this time It has the ability to achieve near-zero accuracy in identifying Category 2, while maintaining high accuracy in identifying other categories.

[0099] Step S3, Input data confirmation - Logits distribution alignment based on knowledge distillation constraints - Generation of incremental model to suppress catastrophic memories:

[0100] Select the remaining category data (excluding the target forgotten category) from the original dataset as incremental learning data and add a quantitative (e.g., 10%) category 2 data as input data to simulate the actual scenario. Use the generated forgetting model as the teacher model to initialize the student model. The student model constrains the output distribution on the input data through knowledge distillation loss and updates the parameters of the student model.

[0101] Furthermore, such as Figure 2 As shown in the incremental learning phase, S3 specifically includes:

[0102] S31. Select the remaining category data from the original dataset as incremental learning data and add a quantitative amount of target forgotten category data as input data for incremental learning;

[0103] S32. Using the forgetting model As a teacher model, initialize the student model. This allows it to inherit the convolutional layer parameters of the teacher model, selectively initialize the fully connected layer parameters, and then perform knowledge distillation training.

[0104] S33. In each training batch, for the new data x learned during the incremental learning process, calculate the output of the teacher model. The output logits of the teacher model is defined as... :

[0105] (4);

[0106] In each training batch, for each new data x learned during incremental learning, the output of the student model is calculated. The output logits of the student model is defined as follows: :

[0107] (5);

[0108] S34. Calculate the knowledge distillation loss. The teacher model uses soft labels to guide the student model in maintaining a forgotten state. (Knowledge distillation loss is then calculated.) Defined as:

[0109] (6);

[0110] In formula (6), T is a temperature parameter used to smooth the output distribution. The role of the temperature parameter T is to make the output distribution of the teacher model smoother, thus making it easier for the student model to learn.

[0111] S35. Calculate the total loss function, combining cross-entropy loss and knowledge distillation loss:

[0112] (7);

[0113] In formula (7), α is a hyperparameter used to balance the weights of the two losses.

[0114] S36, Total Losses Due to Use Perform backpropagation and update the parameters of the student model:

[0115] (8);

[0116] In formula (8), η is the learning rate. This represents the set of trainable parameters for the student model, which are gradually updated through optimization during the incremental learning phase. This indicates that the loss function applies to the student model parameters. The gradient.

[0117] S37. Set up custom iteration rounds. After each training round, verify the overall loss and accuracy of the model on the test set. At the same time, extract the test accuracy and loss for the target forgotten category. If the test accuracy and loss for the target forgotten category in the current round are better than the previously recorded best accuracy and loss, then update the best model state. After all training rounds, save the student model state with the highest test accuracy as the final model.

[0118] Thus, we have obtained an incremental model, the final model, that can both learn new knowledge and suppress forgetting categories, to ensure the persistence of the forgetting effect.

[0119] Furthermore, this embodiment applies the method to medical data security. An initial diagnostic model is trained on a full-category medical dataset. A forgetting model is generated by performing target category suppression on the patient data to be forgotten. Incremental medical data and a small amount of deleted patient data (inevitably some data will be left behind, as well as data that is extremely similar to the forgotten data) are then integrated. A catastrophic recall immune diagnostic model is generated by aligning the Logits distribution with knowledge distillation constraints.

[0120] The experimental proof process is as follows:

[0121] This invention was tested on the MNIST dataset, a 10-class dataset consisting of 60,000 training images and 10,000 test images. These images are grayscale images, each with a resolution of 1x28x28 pixels. The experiments were conducted on a Lenet-5 convolutional neural network, a classic neural network learning architecture primarily used for handwritten digit recognition tasks.

[0122] Since the process of this invention involves a series of tasks, from training an initial model, forgetting data points to obtain a forgotten model, and then incrementally learning to obtain a new model, the results of this invention will be presented step by step. All experiments in this invention involve removing category 2 from the dataset. First, this invention selects 50% of the data from all categories to obtain an initial model. Then, this invention uses two forgetting methods to forget category 2. In the subsequent incremental learning, this invention learns the remaining 50% of the data from other categories and extracts a small amount of data from category 2 for learning. Then, this invention observes the changes in the model.

[0123] like Figure 3As shown, this illustrates how the classification accuracy of the present invention changes with the number of training epochs on the test set when trained on the MNIST dataset using the Lenet-5 model. The horizontal axis represents the number of training epochs, totaling 10 epochs, while the vertical axis represents the accuracy (%), ranging from 0% to 100%. The figure shows that the model's accuracy on the test set gradually improves with the increase in training epochs. The accuracy is approximately 92% in the initial stage (2 training epochs), steadily increasing with each epoch, eventually approaching 100% after 10 training epochs. This demonstrates that the Lenet-5 model performs well on the MNIST dataset, effectively learning and improving its classification performance.

[0124] Then, this invention employs two different forgetting learning methods: one is "amnesia-based forgetting learning," and the other is the KL divergence fitting method. For example... Figure 4 As shown, the model's accuracy varied across different training epochs after applying the "amnesia-based forgetting learning" method. The graph contains two curves: one representing the accuracy of the target class (the model's accuracy in identifying the data category to be forgotten or deleted), and the other representing the accuracy of the model in identifying non-target class data. Before the data removal request, the accuracy of both the target and other classes was relatively high with minimal fluctuations. After the forgetting request, the accuracy of the target class dropped rapidly and approached zero, indicating that amnesia-based forgetting learning effectively eliminated the influence of the target class data in the model.

[0125] like Figure 5 As shown, after applying the KL divergence forgetting method, the accuracy of the target class and other classes was relatively high before the data removal request. After the data removal request, the accuracy of the target class dropped rapidly due to the influence of the KL divergence, while the accuracy of other classes was also affected to some extent by changes in model parameters. After three rounds of adjustment to the KL divergence of the target class and simple data fine-tuning in two rounds, the model accuracy of other classes recovered, and the target class remained stably in a forgotten state.

[0126] After incrementally learning the remaining 50% of the data from other categories and a small amount of data from category two, the results are shown in Tables 1 and 2 (normal incremental learning section). The confusion matrix is ​​used to evaluate the model's performance on the dataset and shows a comparison between the model's predictions for each category and the actual labels. The rows of the matrix represent the actual categories, the columns represent the predicted categories, and the number in each cell indicates the number of samples predicted as that category. These two figures show that the accuracy of the forgotten category two is already very high after incremental learning, indicating that the forgotten model's performance on the forgotten category improves rapidly after incremental learning due to the mixing of a small amount of removed data.

[0127] By incorporating knowledge distillation into the incremental learning process, the incremental learning results shown in Tables 1 and 2 based on knowledge distillation are very significant. Although the forgotten model incorporates a small amount of forgotten category data during incremental learning, the model is subject to feature suppression from the distillation model during the incremental process. This results in the model maintaining a low accuracy (close to 0%) on the forgotten category after incremental learning, while the accuracy on other categories remains unchanged. This reflects the effectiveness of the method of this invention.

[0128] Table 1. Experimental results showing the change in accuracy of Category 2 with training epochs for the forgetting model after "amnesia" and with the incremental learning method of this invention.

[0129]

[0130] Table 2 shows the experimental results of the accuracy of the forgetting model after KL divergence forgetting, normal incremental learning, and incremental learning of this invention for category 2 as a function of training epochs.

[0131]

[0132] Example 2

[0133] This embodiment provides an apparatus for implementing a method for resolving catastrophic recall in forgetting learning based on knowledge distillation. The apparatus includes:

[0134] At least one processor; and,

[0135] A memory that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the knowledge distillation-based method for resolving catastrophic recall in forgetting learning as described above.

[0136] In this embodiment, electronic devices include, but are not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smartphones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, etc.

[0137] Example 3

[0138] This embodiment also provides a computer-readable storage medium storing executable instructions that, when executed, cause the machine to perform the method described above for resolving catastrophic recall in forgetting learning based on knowledge distillation.

[0139] Specifically, a system or apparatus equipped with a readable storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system or apparatus can read and execute the instructions stored in the readable storage medium.

[0140] In this case, the program code itself, which can be read from the readable medium, can perform the functions of any of the above embodiments, and therefore the computer-readable code and the readable storage medium storing the computer-readable code constitute a part of this specification.

[0141] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0142] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A method for resolving catastrophic recall in learning and forgetting based on knowledge distillation, characterized in that, The method includes: S1. Select an initial model and train it using a standard dataset, optimizing the initial model parameters by minimizing the cross-entropy loss; S2. Select the target forgetting category, apply the forgetting-like method to adjust the model parameters, and generate a forgetting model whose ability to identify the target forgetting category is lower than its accuracy in identifying other categories. S3. Select the remaining category data (excluding the target forgetting category) from the original dataset as incremental learning data and add a quantitative amount of target forgetting category data as input data. Use the generated forgetting model as the teacher model to initialize the student model. The student model constrains the output distribution on the input data through knowledge distillation loss and updates the parameters of the student model. S1 specifically includes: S11. Select a standard image dataset as training data; S12. Select a convolutional neural network architecture as the initial model. ; S13. Train the initial model using each category of the dataset. This enhances its ability to identify each category. The training process optimizes the initial model parameters by minimizing the cross-entropy loss (LCE). （1）； In formula (1), It is the input data. These are the actual labels corresponding to the input data. It is the real label number Each component is a category. The tag value, It is the total number of categories. It is the initial model For input data Category The predicted probability; S2 specifically includes: S21. Select a specific category from the dataset as the target category for forgetting, and then perform the forgetting operation on this category. S22. Simultaneously apply two forgetting methods: one is amnesia-based forgetting learning, and the other is the KL divergence forgetting method; Amnesia-based learning method: Record each mini-batch of samples and its corresponding parameter updates during training. When a request to forget a specific category is received, backtrack the parameter updates of the batch including samples of the specific category, thereby removing the learning traces of the data of the forgotten target category. The KL divergence forgetting method involves selecting another dataset and choosing a sample category from this dataset whose similarity to the forgotten target category is greater than a set threshold. The number of sample categories is the same as the number of forgotten target categories. By fitting the KL divergence, the sample categories are aligned with the output distribution of the target category, further reducing the model's ability to identify the forgotten target category.

2. The method for solving catastrophic recall in forgetting learning based on knowledge distillation according to claim 1, characterized in that, The aforementioned amnesia-based learning specifically includes: (1) Record the training process: during model training During the process, it records whether each batch includes data of a specific category, and the corresponding parameter update values ​​for each batch. , Indicates the first The round and the If the parameter update amount in a batch does not include data of a specific category, skip this step. (2) Identify batches of data in a specific category: Find all batches of data in a specific category from the records. , It is a collection that includes batch indexes of all data of a specific category; (3) Undo parameter updates related to a specific category: from the model's current parameters Subtract the updated values ​​of parameters related to a specific category from the middle, that is: （2）； In formula (2), These are the parameters of the model after training. These are the model parameters after the forgetting operation. It is the sum of parameter update values ​​for all batches that include data of a specific category.

3. The method for resolving catastrophic recall in forgetting learning based on knowledge distillation according to claim 2, characterized in that, The KL divergence forgetting method specifically includes: (11) Using the initial model Predict the output distribution of a specific target category and the output distribution of the corresponding sample categories in another dataset. ; (22) Calculate the KL divergence to measure the probability distributions of two probability distributions. and Differences between them: （3）； In formula (3), It is the predicted probability of the i-th class output by the initial model M when the input is a sample of a specific class of the target; It is the predicted probability of the i-th class output by the initial model M for sample classes with the same semantics as the target class; (33) By minimizing the KL divergence, the model parameters are adjusted to force the model to distribute the output of the target specific class. Distribution to QMNIST Alignment eliminates unique features that define a specific category; (44) Exclude the training data of the forgotten class, extract a small amount of training data from the remaining original training data and set a learning rate threshold and iteration rounds, and fine-tune it using the same optimizer as the original training; at the same time, during the fine-tuning process, the model calculates the loss function through forward propagation and updates the parameters through back propagation. Initial model The forgetting model was obtained through the two forgetting methods described above. .

4. The method for resolving catastrophic recall in forgetting learning based on knowledge distillation according to claim 3, characterized in that, S3 specifically includes: S31. Select the remaining category data from the original dataset as incremental learning data and add a quantitative amount of target forgotten category data as input data for incremental learning; S32. Using the forgetting model As a teacher model, initialize the student model. It inherits the convolutional layer parameters from the teacher model, selectively initializes the fully connected layer parameters, and then performs knowledge distillation training. S33. In each training batch, for the new data x learned during the incremental learning process, calculate the output of the teacher model. The output logits of the teacher model is defined as... : （4）； In each training batch, for each new data x learned during incremental learning, the output of the student model is calculated. The output logits of the student model is defined as follows: : （5）； S34. Calculate the knowledge distillation loss. The teacher model uses soft labels to guide the student model in maintaining a forgotten state. (Knowledge distillation loss) Defined as: （6）； In formula (6), T is a temperature parameter used to smooth the output distribution. The role of the temperature parameter T is to make the output distribution of the teacher model smoother, so that it is easier for the student model to learn. S35. Calculate the total loss function, combining cross-entropy loss and knowledge distillation loss: （7）； In formula (7), α is a hyperparameter used to balance the weights of the two losses; S36, Total Losses Due to Use Perform backpropagation and update the parameters of the student model: （8）； In formula (8), η is the learning rate. This represents the set of trainable parameters for the student model, which are gradually updated through optimization during the incremental learning phase. This indicates that the loss function applies to the student model parameters. The gradient; S37. Set up custom iteration rounds. After each training round, verify the overall loss and accuracy of the model on the test set. At the same time, extract the test accuracy and loss for the target forgotten category. If the test accuracy and loss for the target forgotten category in the current round are better than the previously recorded best accuracy and loss, then update the best model state. After all training rounds, save the student model state with the highest test accuracy as the final model.

5. The method for solving catastrophic recall in forgetting learning based on knowledge distillation according to claim 1, characterized in that, The tag is one-hot encoded.

6. An apparatus for implementing a method for resolving catastrophic recall in forgetting learning based on knowledge distillation, characterized in that, The device includes: processor; A memory on which computer programs that can run on the processor are stored; When the computer program is executed by the processor, it implements the steps of the method for solving catastrophic memory problems in learning based on knowledge distillation as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images