Neural network model denoising method based on knowledge migration
By adopting a method of combining knowledge transfer and differential information in deep neural networks, using loss function training of teachers and students' models, the problem of noise robustness to deep neural networks is solved, and the higher level is achieved in different noise environments accuracy and robustness.
Patent Information
- Application Number
- CN202411269558.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-12-27
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When the prior art deals with deep neural networks under the influence of noise, it is difficult to effectively enhance the robustness of network weights during training, resulting in a greater impact on system functions. The noise characteristics and accuracy are not discussed from the perspective of representation learning. relationship.
Using a neural network model denoising method based on knowledge transfer, combining mutual information differences with knowledge distillation, teachers and students' neural network models are constructed, and the first-stage and second-stage loss functions are used to train in different noise environments to ensure that the student model is noise-free Maintain high accuracy under conditions.
The accuracy and robustness of the neural network are significantly improved under different degrees of noise. With the increase in noise level, the accuracy is higher, which enhances the anti-noise ability of the neural network.
Smart Images
Figure CN119204090A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the fields of information theory and neural network technology, and in particular to a neural network model denoising method based on knowledge transfer. Background Art
[0002] Noise is inherent in almost all physical media, phenomena, and operations. Therefore, noisy neural networks often appear in various application scenarios.
[0003] When a deep neural network is deployed on a noisy medium, the analog device is susceptible to the effects of its physical components (such as temperature, random telegraph, or pressure), which ultimately cause the deployed weights to be different from the expected weights on the analog device. That is, analog computing is inherently noisy, and the neural network deployed on it is a noisy neural network. This situation is usually the case where the weights of the trained neural network are mapped to the memristor crossbar array of the memristor, the calculations inside the deep neural network are performed in the analog domain, and the digital weights are represented by analog quantities such as resistance or voltage. It is worth noting that the noise in analog hardware is usually modeled as Gaussian additive white noise to simulate the overall impact of many concurrent random effects.
[0004] When deep neural networks are transmitted through wireless channels or when deep neural network parameters are truncated or quantized to adapt to limited system accuracy, deep learning applications often require multiple agents to perform distributed learning and reasoning. In such systems, wireless communication is an important transmission method. However, wireless channels introduce additive Gaussian white noise to the transmitted signals, which ultimately affects the effectiveness of the system.
[0005] Therefore, how to enhance the robustness of network weights so that the impact of noise weights on the overall system function is minimized is the main research direction now. For example, noise can be introduced during the training process to make it more noise-resistant in the inference phase; or a robust noisy neural network can be obtained in a noise-free training environment. The former, such as the hardware-aware noise training method, introduces noise injection training in the scenario where the neural network is mapped to RRAM to improve the noise robustness of the model in the inference phase, or combines noise training with the batch normalization layer to batch normalize the input of the activation layer. The latter uses L2 regularization and error compensation based on the model to deal with weight noise, or only in the prediction phase, batch normalization is performed on the input of the softmax layer, so that the accuracy of the model is improved compared to before.
[0006] However, none of the above methods considers the relationship between noise characteristics and accuracy from the perspective of representation learning. Therefore, in order to solve this problem, the present invention proposes a denoising method for a neural network model based on knowledge transfer. Summary of the invention
[0007] The purpose of the present invention is to provide a neural network model denoising method based on knowledge transfer, which combines mutual information difference with knowledge distillation and uses noise training to maintain a higher accuracy under different degrees of noise. The accuracy is higher with the increase of the noise level, which is beneficial to enhancing the noise robustness of the neural network and has obvious advantages.
[0008] To achieve the above object, the present invention provides a neural network model denoising method based on knowledge transfer, which is characterized by comprising the following steps:
[0009] S1. Construct a teacher neural network model and a student neural network model with the same structure as the teacher neural network model according to the actual situation;
[0010] S2. Use the one-stage loss function L p1 Calculate the loss between the output of the student neural network model and the output of the teacher neural network model;
[0011] S3, judging whether the training has converged, if it has, freezing the parameters of the feature extractor of the student neural network model, and no longer updating the parameters; otherwise, repeating step S2 until the model converges;
[0012] S4. Use a two-stage loss function Calculate the loss between the output of the student neural network model and the output of the teacher neural network model;
[0013] S5. Determine whether the model has converged. If so, the model training is completed and the results are output; otherwise, repeat step S4 until convergence.
[0014] Preferably, in step S1, according to actual conditions, a teacher neural network model is constructed, consisting of an upstream feature extractor and a downstream task classifier, which is trained until convergence under clean weights that are not affected by noise and set to the inference stage, and the parameters of the teacher neural network model are fixed; at the same time, a student neural network model with the same structure as the teacher neural network model is initialized.
[0015] Preferably, in step S2, noise is injected into the weights of the student neural network model during training, and the training data set is simultaneously input into the teacher neural network model and the student neural network model to obtain the outputs of the feature extractors and classifiers of the two models.
[0016] Preferably, a one-stage loss function is used The loss between the student neural network model output and the teacher neural network model output is calculated as follows:
[0017]
[0018] in, is the hard loss, which is the loss between the output of the student neural network model and the true label; It is a soft loss, which is calculated by the difference between the output of the teacher neural network model and the student neural network model; is the L2 regularization term; W S and is the neural network weight parameter; x is the input data.
[0019] Preferably, hard loss It is used to ensure that the student neural network model can not only imitate the behavior of the teacher neural network model, but also predict the true label, as shown below:
[0020]
[0021] Among them, α is a hyperparameter, is the cross entropy loss function, is the output of the i-th input of the student neural network model, y true is the one-hot encoding of the true label;
[0022] In the process of knowledge distillation, soft loss It is used to measure the similarity between the output of the student neural network model and the output of the teacher neural network model, as follows:
[0023]
[0024] Among them, β1, β2 are hyperparameters, D KL is the KL divergence, Corresponding to the output of the i-th input of the student neural network model and the teacher neural network model respectively; DMI δ represents the mutual information difference, which is defined as follows:
[0025] DMI δ =I(X;g(X))-I(X;g n (X)
[0026] Where X represents the input, and D JS (g,g n ) = δ, g(X) and g n (X) are the output of the teacher neural network model extractor and the output of the student neural network model feature extractor, respectively; the hyperparameters α, β1, β2 are used to balance the weight between soft loss and hard loss.
[0027] Preferably, in step S4, noise training is performed on the weights of the student neural network model, and the training data set is simultaneously input into the teacher neural network model and the noisy student neural network model to obtain classifier outputs of the two models.
[0028] Preferably, a two-stage loss function is used The loss between the student neural network model output and the teacher neural network model output is calculated as follows:
[0029]
[0030] Among them, soft loss is defined as follows:
[0031]
[0032] Among them, during the two-stage training process, the feature extractor is already robust and does not need to be trained.
[0033] Therefore, the present invention adopts the above-mentioned neural network model denoising method based on knowledge transfer, combines mutual information difference with knowledge distillation, and uses noise training to maintain a higher accuracy rate under different degrees of noise. The accuracy is higher with the increase of the noise level, which is beneficial to enhancing the noise robustness of the neural network and has obvious advantages.
[0034] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 This is an overall flow chart of a mutual information driven knowledge transfer and neural network anti-noise technology of the present invention;
[0036] Figure 2 A schematic diagram of a first-stage training architecture in a mutual information-driven knowledge transfer and neural network anti-noise technology of the present invention;
[0037] Figure 3 A schematic diagram of a two-stage training architecture in a mutual information driven knowledge transfer and neural network anti-noise technology of the present invention;
[0038] Figure 4 This is a diagram showing the experimental results of the MNIST data set under low noise in an embodiment of the mutual information driven knowledge transfer and neural network anti-noise technology of the present invention;
[0039] Figure 5 This is a diagram showing the experimental results of the MNIST data set under moderate noise in an embodiment of the mutual information driven knowledge transfer and neural network anti-noise technology of the present invention;
[0040] Figure 6 A diagram of experimental results of the MNIST data set under high noise in an embodiment of a mutual information driven knowledge transfer and neural network anti-noise technology of the present invention. DETAILED DESCRIPTION
[0041] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.
[0042] like Figure 1 As shown, the present invention provides a neural network model denoising method based on knowledge transfer, comprising the following steps:
[0043] S1. Construct a teacher neural network model and a student neural network model with the same structure as the teacher neural network model according to actual conditions.
[0044] According to the actual situation, a teacher neural network model is constructed, which consists of an upstream feature extractor and a downstream task classifier. It is trained until convergence under clean weights that are not affected by noise and set to the inference stage. The parameters of the teacher neural network model are fixed. At the same time, a student neural network model with the same structure as the teacher neural network model is initialized.
[0045] S2, during training, inject noise into the weights of the student neural network model, input the training data set into both the teacher neural network model and the student neural network model, and obtain the outputs of the feature extractors and classifiers of the two models. Figure 2 As shown, a one-stage loss function is used The loss between the student neural network model output and the teacher neural network model output is calculated as follows:
[0046]
[0047] in, is the hard loss, which is the loss between the output of the student neural network model and the true label; It is a soft loss, which is calculated by the difference between the output of the teacher neural network model and the student neural network model; is the L2 regularization term; W S and is the neural network weight parameter; x is the input data.
[0048] Hard Loss It is used to ensure that the student neural network model can not only imitate the behavior of the teacher neural network model, but also correctly predict the true label, as shown below:
[0049]
[0050] Among them, α is a hyperparameter, is the cross entropy loss function, is the output of the i-th input of the student neural network model, y true is the one-hot encoding of the true label.
[0051] In the process of knowledge distillation, soft loss It is used to measure the similarity between the output of the student neural network model (through soft labels) and the output of the teacher neural network model, as shown below:
[0052]
[0053] Among them, β1, β2 are hyperparameters, D KL is the KL divergence, Corresponding to the output of the i-th input of the student neural network model and the teacher neural network model respectively; DMI δ represents the mutual information difference, which is defined as follows:
[0054] DMI δ =I(X;g(X))-I(X;g n (X)
[0055] Where X represents the input, and D JS (g,g n ) = δ, g(X) and g n (X) are the output of the teacher neural network model extractor and the output of the student neural network model feature extractor respectively.
[0056] In addition, the hyperparameters α, β1, β2 are used to balance the weight between soft loss and hard loss.
[0057] S3, judging whether the training has converged, if so, freezing the parameters of the feature extractor of the student neural network model, and not updating the parameters subsequently; if not, repeating step S2 until the model converges;
[0058] S4, inject noise into the weights of the student neural network model for training, input the training data set into the teacher neural network model and the noisy student neural network model at the same time, and obtain the classifier outputs of the two models. Figure 3 As shown, a two-stage loss function is used The loss between the student neural network model output and the teacher neural network model is calculated as follows:
[0059]
[0060] Among them, soft loss is defined as follows:
[0061]
[0062] During the second stage of training, the feature extractor is already robust and no longer needs to be trained, so it is removed from the loss function.
[0063] S5. Determine whether the model has converged. If so, the model training is completed and the results are output; otherwise, repeat step S4 until convergence.
[0064] Example
[0065] This embodiment combines simulation results to illustrate the performance of the proposed solution of the present invention. The simulation conditions are as follows: based on the PyTorch framework on a computer with an NVIDIA GeForce RTX 3060 GPU. A set of model architectures of the same size is selected in the simulation, and the noise type selected is Gaussian additive white noise. The injection method is to add the noise independently to the model weight in each round of training or testing, and three groups of Gaussian white noise with different standard deviations are used for accuracy testing. The three groups of standard deviations range from (0.1, 0.2, 0.3), (0.4, 0.5, 0.6), (0.7, 0.8, 0.9); the LeNet5 model is used to start training from the pre-trained model on the MNIST data set, and the training batch size is set to 64; in the first stage of training, the hyperparameters α, β1, and β2 are set to 0.5, 0.75, and 0.8 respectively; in the second stage of training, the hyperparameters α and β1 are set to 0.6 and 0.6 respectively; the MINE neural network estimator is used to calculate the mutual information difference in the loss function.
[0066] In order to compare the accuracy of the mutual information driven knowledge transfer method proposed in the present invention with the existing noise training method and batch normalization regularization under the same conditions, Figure 4 , Figure 5 and Figure 6 As shown, the experimental results of the existing method and the method are respectively shown in the LeNet5 model using the MNIST data set under three levels of noise. In the experiment, four different methods are used, a clean model without any processing, a noise training method, a batch normalization denoising method, and a neural network model denoising method based on knowledge transfer of the present invention. In comparison, the method proposed by the present invention maintains a higher accuracy rate under different levels of noise, and the accuracy is higher with the increase of the noise level. The superiority of the method proposed by the present invention is more obvious.
[0067] Therefore, the present invention adopts the above-mentioned neural network model denoising method based on knowledge transfer, combines mutual information difference with knowledge distillation, and uses noise training to maintain a higher accuracy rate under different degrees of noise. The accuracy is higher with the increase of the noise level, which is beneficial to enhancing the noise robustness of the neural network and has obvious advantages.
[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.
Claims
1. A neural network model denoising method based on knowledge transfer, characterized in that: The following steps are involved: S1. Construct a teacher neural network model and a student neural network model with the same structure as the teacher neural network model according to the actual situation; S2. Use the one-stage loss function L p1 Calculate the loss between the output of the student neural network model and the output of the teacher neural network model; S3, judging whether the training has converged, if it has, freezing the parameters of the feature extractor of the student neural network model, and no longer updating the parameters; otherwise, repeating step S2 until the model converges; S4. Use the two-stage loss function L p2 Calculate the loss between the output of the student neural network model and the output of the teacher neural network model; S5. Determine whether the model has converged. If so, the model training is completed and the results are output; otherwise, repeat step S4 until convergence.
2. The neural network model denoising method based on knowledge transfer according to claim 1, characterized in that: In step S1, according to the actual situation, a teacher neural network model is constructed, which consists of an upstream feature extractor and a downstream task classifier. It is trained until convergence under clean weights that are not affected by noise and set to the inference stage. The parameters of the teacher neural network model are fixed. At the same time, a student neural network model with the same structure as the teacher neural network model is initialized.
3. The neural network model denoising method based on knowledge transfer according to claim 1, characterized in that: In step S2, noise is injected into the weights of the student neural network model during training, and the training data set is simultaneously input into the teacher neural network model and the student neural network model to obtain the outputs of the feature extractors and classifiers of the two models.
4. The neural network model denoising method based on knowledge transfer according to claim 3 is characterized in that: Using a one-stage loss function The loss between the student neural network model output and the teacher neural network model output is calculated as follows: in, is the hard loss, which is the loss between the output of the student neural network model and the true label; It is a soft loss, which is calculated by the difference between the output of the teacher neural network model and the student neural network model; is the L2 regularization term; W S and is the neural network weight parameter; x is the input data.
5. The neural network model denoising method based on knowledge transfer according to claim 4 is characterized in that: Hard Loss It is used to ensure that the student neural network model can not only imitate the behavior of the teacher neural network model, but also predict the true label, as shown below: Among them, α is a hyperparameter, is the cross entropy loss function, is the output of the i-th input of the student neural network model, y true is the one-hot encoding of the true label; In the process of knowledge distillation, soft loss It is used to measure the similarity between the output of the student neural network model and the output of the teacher neural network model, as follows: Among them, β1, β2 are hyperparameters, D KL is the KL divergence, Corresponding to the output of the i-th input of the student neural network model and the teacher neural network model respectively; DMI δ represents the mutual information difference, which is defined as follows: DMI δ =I(X;g(X))-I(X;g n (X)) Where X represents the input, and D JS (g,g n ) = δ, g(X) and g n (X) are the output of the teacher neural network model extractor and the output of the student neural network model feature extractor, respectively; the hyperparameters α, β1, β2 are used to balance the weight between soft loss and hard loss.
6. The neural network model denoising method based on knowledge transfer according to claim 3, characterized in that: In step S4, noise training is performed on the weights of the student neural network model, and the training data set is simultaneously input into the teacher neural network model and the noisy student neural network model to obtain the classifier outputs of the two models.
7. The neural network model denoising method based on knowledge transfer according to claim 6, characterized in that: Using a two-stage loss function The loss between the student neural network model output and the teacher neural network model output is calculated as follows: Among them, soft loss is defined as follows: Among them, during the two-stage training process, the feature extractor is already robust and does not need to be trained.