Distillation online training circuit and method for realizing fault tolerance function of neural network

Abstract

The invention belongs to the field of artificial neural networks, and particularly relates to a distillation online training circuit and method for realizing a neural network fault tolerance function. The circuit comprises a student model neural network model circuit, a teacher model neural network model circuit and a knowledge distillation training loss calculation module, wherein the student model neural network model circuit and the teacher model neural network model circuit are symmetrically distributed; the student model neural network model circuit and the teacher model neural network model circuit are formed by connecting convolution layers and full connection layers, the last full connection layer is a classification layer and is in output connection with a Softmax activation function circuit, the other full connection layers and all convolution layers are in output connection with a ReLU activation function circuit, and the output of each layer after activation forms an output feature map; the knowledge distillation training loss calculation module comprises a characteristic distillation loss technology module, a response distillation loss calculation module and a cross entropy loss calculation module. According to the method, the cost is lower during fault detection, and the detection efficiency is higher.

Description

Technical Field

[0001] The present invention belongs to the field of artificial neural networks, and in particular relates to a distillation online training circuit and method for realizing the fault tolerance function of a neural network. Background Art

[0002] As a new type of non-volatile memory, memristors possess properties similar to biological synapses, enabling them to achieve brain-like learning. Due to their advantages such as low power consumption and high integration, memristors are widely used in hardware implementations of various neural networks. Deep neural networks (DNNs) have become an important tool for solving complex problems and are used to handle various complex tasks such as image recognition and big data analysis. DNN hardware architectures have also been extensively studied. Integrated storage and computing overcomes the bottleneck of the von Neumann architecture, which requires frequent data transfer between data storage and computing units, by performing computations directly in the storage unit. This makes it one of the most effective approaches to implementing DNN hardware architectures.

[0003] The rapid development of deep learning technology has led to an increase in the size and complexity of models, resulting in increasing demands for computing resources and energy efficiency. Knowledge distillation, as an effective model compression technique, has been widely used to improve the performance of small models by migrating knowledge from complex models to smaller models, thereby achieving high-precision inference at lower computational and storage costs. Traditional knowledge distillation relies on offline training on general-purpose computing platforms, which struggles to meet the real-time performance and low power requirements of edge devices. Developing hardware architectures specifically for knowledge distillation can transform the theoretical advantages of distillation algorithms into energy efficiency improvements at the hardware level.

[0004] Due to the immaturity of the manufacturing process and the non-ideality of memristor devices, memristors are prone to stuck failures. This causes memristor-based computing systems to frequently experience non-ideal problems and be unable to perform accurate calculations. Efficient solutions are needed to solve these problems in order to improve the fault tolerance of edge devices. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a distillation online training circuit and method for realizing the fault tolerance function of a neural network, that is, to provide a complete fault detection and knowledge distillation online training solution on an edge device, which solves the problem that existing edge devices are difficult to meet the requirements of low power consumption and real-time knowledge distillation training, efficiently solves the stuck fault of the memristor in the memristive neural network, realizes rapid fault location in a large-area memristive cross array, and uses a self-compensation solution to solve the pruning problem of faulty memristors in the memristive cross array.

[0006] The technical solutions adopted are:

[0007] A circuit for implementing distillation online training with a fault-tolerant neural network function includes a student model neural network model circuit (referred to as the student model, the same below), a teacher model neural network model circuit (referred to as the teacher model, the same below), and a knowledge distillation training loss calculation module. The student model neural network model circuit and the teacher model neural network model circuit are symmetrically distributed.

[0008] The neural network model circuits of both the student model and the teacher model include several convolutional layers and several fully connected layers, labeled layer 1, layer 2, ... layer n, where n is a natural number greater than 1; the last fully connected layer is the classification layer, and its output is connected to the Softmax activation function circuit. The outputs of the remaining fully connected layers and all convolutional layers are connected to the ReLU activation function circuit, and then connected to the output feature map;

[0009] The knowledge distillation training loss calculation module includes the feature distillation loss technology module, the response distillation loss calculation module, and the cross entropy loss calculation module;

[0010] The output feature map of each layer in the student model is compared in voltage form with the output feature map of the corresponding layer in the teacher model. The student model is aligned with the features of the intermediate layer by minimizing the feature distillation loss calculation module. At the classification layer, the probability distribution of the student model's output is compared with the output probability distribution of the teacher model. The student model learns the teacher model's output by minimizing the response distillation loss calculation module. Similarly, at the classification layer, the student model's output probability distribution is simultaneously compared with the true label, and the student model learns the true label by minimizing the cross-entropy loss. Three loss functions (feature distillation loss, response distillation loss, and cross-entropy loss) jointly supervise the student model's online training, enabling knowledge distillation to run efficiently on edge devices. The intermediate layer is the interconnected teacher model (or student model). The first layer is called the input layer, the last layer is called the output layer (classification layer), and each layer between the input and output layers is called an intermediate layer.

[0011] Preferably, the output feature map of each layer in the student model and the output feature map of the corresponding layer in the teacher model are connected to the feature distillation loss calculation module, and are calculated and compared through the feature distillation loss calculation module.

[0012] A distillation online training method for realizing a neural network fault tolerance function comprises the following steps:

[0013] (1) Apply a voltage exceeding the threshold to all memristors and initialize the resistance to R min , then divide the complete memristor crossbar array into at least 2 regions evenly, and write the same change △R to all memristors;

[0014] (2) Apply a test voltage from the input port to each test area and calculate the ideal output result as a reference voltage; set a shared current sensor with a TIA structure at the column port of the memristor crossbar array and compare the output result with the reference voltage. If the comparison result has no error, it means there is no fault; if the result has an error, it means that there is a stuck fault in the detection area;

[0015] If the determined range exceeds the maximum limit, the divided area is further divided and further tested using the above method to narrow the fault area. The final fault area is determined and the memristors in the area are marked.

[0016] (3) The array is retrained using the designed multi-loss knowledge distillation training architecture, and the back-propagation gradient of the weights is calculated. According to the previously marked memristors, if they are marked memristors, the self-compensation method is used for pruning. If they are unmarked memristors, normal updates are performed according to the gradient calculation formula to complete the retraining.

[0017] Preferably, when the memristors in the memristive neural network are fault-free, the LeNet-5 model and the AlexNet model constructed using the memristive crossbar array can achieve accuracies of 98.67% and 76.29% on the MNIST dataset and the CIFAR-10 dataset; when the memristors are defective, the reasoning accuracy of the two neural network models constructed by the memristive crossbar array on the two datasets is impaired and cannot meet the accuracy requirements.

[0018] Preferably, the memristor crossbar array is divided into at least two regions, and fault detection is performed on each region, instead of the traditional single memristor detection, the same change ΔR is written to all memristors, and then a test voltage V is applied to each test region from the input port. test , calculate and output the test result △R×V test ; Use the designed fault detection circuit to compare the output result with the set reference voltage through the static voltage comparison method. If there is a difference with the ideal situation, it is determined to be a fault, otherwise it is considered to be no fault

[0019] Preferably, the teacher model neural network model circuit and the student model neural network model circuit based on the memristor cross array use the 2M method to represent signed weights, the signed weights are represented by the difference in the conductance of a pair of memristors, and the normal memristor in a pair of memristors is used to compensate for the faulty memristor, and the faulty weights are pruned by adjusting the conductance of the normal memristor in the memristor pair to be the same as or close to the conductance of the faulty memristor.

[0020] Preferably, since a weight is represented by the difference in conductance between a pair of memristors, the situations when one or both of the pair of memristors fail are different; for different situations, the adjustable range of conductance is different, and the weight mapping range also changes, and the faulty memristor is compensated according to the mapping range.

[0021] Preferably, the pruned network architecture is retrained using a knowledge distillation training loss calculation module, and the online training architecture can be performed directly on the edge device without relying on offline devices.

[0022] Preferably, the multi-loss knowledge distillation online training of the pruned student model is implemented on the circuit architecture according to the back-propagation principle; the output of the pruned student model neural network model circuit is compared with the output of the teacher model neural network model circuit, so that the student model can still capture the global information and inter-class relationship of the data after pruning, and the output of the middle layer of the teacher model neural network model circuit is used as the knowledge to supervise the training of the student model neural network model circuit, and the unpruned middle layer feature knowledge is complementary to the response knowledge.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] (1) A fault detection method based on parallel detection by region is adopted, and the sparsity of the neural network itself is used to select the range for pruning. Compared with the inefficient traditional single memristor fault sequential detection, this fault detection method takes less time cost and has higher detection efficiency.

[0025] (2) The selected fault range is pruned, and a self-compensating pruning strategy is adopted to solve the problem that the faulty memristor is difficult to remove on the integrated edge device. Through different weight mappable ranges, the faulty memristor is compensated by the compensating memristor to achieve pruning. This can solve the problem of weight value drift caused by stuck faults and further alleviate the serious degradation of network inference performance.

[0026] (3) A multi-loss knowledge distillation online training architecture is designed, which can implement error backpropagation for distillation training based on in-situ learning. The three loss functions are used to jointly supervise the training. It can be performed in real time with low power consumption on edge devices, does not rely on offline training on general computing platforms, and can be used for retraining after pruning faulty memristors on a memristor crossbar array to compensate for the accuracy loss caused by pruning a large amount of weights. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of a circuit architecture for online distillation training that implements fault-tolerant neural networks.

[0028] Figure 2This is a flow chart of a distillation online training method for realizing a neural network fault tolerance function according to the present invention;

[0029] Figure 3 This is a schematic diagram of the basic architecture of a memristor crossbar array with signed weights using the 2M method in the present invention;

[0030] Figure 4 The effects of stuck-at-0 fault and stuck-at-1 fault on the accuracy of the two models under different degrees of the present invention;

[0031] Figure 5 The effect of pruning the two neural network models of the present invention on the recognition accuracy of the two data sets;

[0032] Figure 6 A schematic diagram of the fault detection and fault area determination process of the present invention;

[0033] Figure 7 A schematic diagram of the pruning area determined by the present invention;

[0034] Figure 8 This is a schematic diagram of pruning performed using the half-step voltage method and the self-compensation method of the present invention;

[0035] Figure 9 It is a measure of the change of different loss functions during the knowledge distillation training process of the present invention;

[0036] Figure 10 This is the result of retraining the pruned model using the multi-loss knowledge distillation method of the present invention. DETAILED DESCRIPTION

[0037] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. "Several" or "multiple" is understood in accordance with the notification of technicians in the relevant technical field, that is, it can be 2, 3, 4, 5...100...1000...n, where n is a natural number greater than 1 and is set according to actual needs.

[0038] Example 1

[0039] like Figure 1 As shown, a circuit for implementing distillation online training of a neural network fault tolerance function includes a student model neural network model circuit, a teacher model neural network model circuit, and a knowledge distillation training loss calculation module. The student model neural network model circuit and the teacher model neural network model circuit are symmetrically distributed.

[0040] The neural network model circuits of both the student model and the teacher model include several convolutional layers and several fully connected layers, labeled layer 1, layer 2, layer 3, ... layer n, where n is a natural number greater than 1; the last fully connected layer is the classification layer, and its output is connected to the Softmax activation function circuit. The outputs of the remaining fully connected layers and all convolutional layers are connected to the ReLU activation function circuit, and then connected to the output feature map;

[0041] The knowledge distillation training loss calculation module includes the feature distillation loss technology module, the response distillation loss calculation module, and the cross entropy loss calculation module;

[0042] The output feature map of each layer in the student model is compared with the output feature map of the corresponding layer of the teacher model in the form of voltage, and the student model is aligned with the features of the intermediate layer by minimizing the feature distillation loss calculation module; in the classification layer, the probability distribution of the student model output is compared with the output probability distribution of the teacher model, and the student model learns the output of the teacher model by minimizing the response distillation loss calculation module; similarly, in the classification layer, the output probability distribution of the student model is compared with the true label at the same time, and the student model learns the true label by minimizing the cross-entropy loss; three loss functions (i.e., feature distillation loss function, response distillation loss function, and cross-entropy loss function) jointly supervise the student model for online training, so that knowledge distillation can run efficiently on edge devices.

[0043] The output feature map of each layer in the student model and the output feature map of the corresponding layer in the teacher model are connected to the feature distillation loss calculation module, and are calculated and compared through the feature distillation loss calculation module.

[0044] like Figure 2 As shown, a distillation online training method for realizing the fault tolerance function of a neural network includes the following steps:

[0045] (1) Apply a voltage exceeding the threshold to all memristors and initialize the resistance to R min , then divide the complete memristor crossbar array into at least 2 regions evenly, and write the same change △R to all memristors;

[0046] (2) Apply a test voltage from the input port to each test area and calculate the ideal output result as a reference voltage; set a shared current sensor with a TIA structure at the column port of the memristor crossbar array and compare the output result with the reference voltage. If the comparison result has no error, it means there is no fault; if the result has an error, it means that there is a stuck fault in the detection area;

[0047] If the determined range exceeds the maximum limit, the divided area is further divided and further tested using the above method to narrow the fault area. The final fault area is determined and the memristors in the area are marked.

[0048] (3) The array is retrained using the designed multi-loss knowledge distillation training architecture, and the back-propagation gradient of the weights is calculated. According to the previously marked memristors, if they are marked memristors, the self-compensation method is used for pruning. If they are unmarked memristors, normal updates are performed according to the gradient calculation formula to complete the retraining.

[0049] The present invention proposes a flow chart of the fault-tolerant solution for memristor neural networks based on knowledge distillation online training circuits. Through the three-step innovation of "detection-pruning-retraining", a complete fault-tolerant solution for memristor arrays is constructed. The detection part uses the proposed regional fault detection method, with multiple regions detected in parallel. The fault is located through the designed detection circuit, reducing the time complexity from O(n 2 ) is reduced to O(n 2 / k 2 +k 2 (k is the number of partitions). The pruning phase leverages the advantages of the 2M architecture and uses a self-compensation method to prune faulty memristors, compensating for them with the functioning memristors in the memristor pair. The retraining phase utilizes a designed knowledge distillation online training architecture, which can be performed on edge devices without relying on offline training on general-purpose computing platforms. Knowledge distillation training can offset the adverse effects of extensive weight pruning and restore accuracy to a higher level.

[0050] like Figure 3 Figure 2 shows the basic architecture of a memristor crossbar array with signed weights using the 2M method. First, the memristor crossbar array can change the conductance of the memristors using voltage signals, allowing it to be adjusted to match the weights of the neural network. A vector is input into the array as a voltage, and according to Ohm's law, the multiplication and addition result is obtained as a current, which can be expressed as:

[0051]

[0052] Among them, G ij is the conductance of the memristor synapse in row i and column j, V i is the input voltage of each row, I j Represents the output current of each column. According to Kirchhoff's current law, the output current value of each row naturally summarizes the current flowing through each device, completing the cumulative calculation. Finally, the current value output by each row represents the result vector of the matrix-vector multiplication. Since the weight value has negative values, the 2M method uses the difference between a pair of memristors to represent the signed weight, and the difference in the conductance of two memristors in different columns of the same row (G + -G - ) to achieve positive, negative and zero weights. Using an inverting amplifier, the output current can be expressed as:

[0053]

[0054] in, and Indicates the positive and negative column output current of column j, V j represents the voltage input to the jth column, G Mi+ and G Mi- Represent the conductance values of two memristors in different columns of the same row, and use their difference to represent the signed weight of a synapse.

[0055] Defects generated by memristors during the complex nanoscale manufacturing process, such as oxide layer thickness fluctuations, line edge roughness, feature size shrinkage, etc., will cause memristors to experience stuck-at faults. These stuck-at faults are irreversible, and the memristor is permanently fixed in a high resistance state or a low resistance state. They are named stuck-at-0 fault (SA0) and stuck-at-1 fault (SA1), respectively. This defect can cause serious degradation in the overall performance of computing systems based on memristive crossbar arrays. The present invention uses two neural network models, LeNet-5 and AlexNet, constructed using a memristive crossbar array to identify the MNIST dataset and the CIFAR-10 dataset, respectively, for verification. The basic information of the model in the absence of faults is shown in Table 1. The present invention tests the impact of the two fault types on the accuracy of the two models at different degrees, see Table 1. Figure 4 shown. Figure 4 (a) is the recognition result of the LeNet-5 model on the MNIST dataset. Figure 4 (b) is the recognition result of the AlexNet model on the CIFAR-10 dataset.

[0056] Table 1 Basic information of the model under no fault conditions

[0057]

[0058] Because a weight is represented by the difference in conductance between a pair of memristors, the situations when one or both memristors fail are different. Table 2 shows the situations in which a memristor in a pair fails. The adjustable range of conductance and the range of weight mapping vary with each situation. The table summarizes the various situations in which SA0 and SA1 occur in the memristor pair and the corresponding weight mapping ranges. It can be seen that when a weight needs to be adjusted to 0, situations 1 through 7 provide a good match.

[0059] Table 2 Fault condition combinations and corresponding weight mapping ranges under stuck faults

[0060]

[0061] The present invention uses two neural network models, LeNet-5 and AlexNet, to identify the MNIST dataset and CIFAR-10 dataset respectively for verification. The two trained neural network models are pruned to determine the maximum range that does not affect the operation of the neural network. On this basis, the detected fault range is kept around the maximum range. This can not only maintain accuracy, but also minimize the power consumption of the memristor crossbar array and save detection time. We use Tensorflow to prune the trained neural network, outputting the recognition accuracy every 2% pruning, and determine the sparsity at which the recognition accuracy drops significantly. Figure 5 shown. Figure 5 (a) For the MNIST dataset recognition task, it is confirmed that the overall range of fault pruning can be maintained between 68% and 72%. Figure 5 (b) For the CIFAR-10 dataset task, it is confirmed that the pruning range is between 40% and 42%, at which point the model's accuracy for both datasets drops by no more than 5%.

[0062] like Figure 6 As shown, the complete memristor crossbar array is evenly divided into multiple parts. First, the same change △R is written to all memristors. Then, a test voltage V is applied to each test area from the input port. test , we calculate the output test result △R×V test . A shared current sensor with a TIA structure is set at the column port of the memristor crossbar array, and the output result is compared with the set reference voltage. When the result deviates, it indicates that there is a stuck fault in the detection area. We set the calculation result as the least significant bit (LSB) of the ADC in the detection phase at the output port. A stuck fault will cause one LSB deviation. The cumulative digital output deviation of the column in the selected area is equal to the number of faults. After all areas are tested, the number of memristors in the final fault area is maintained at a certain number. If the total number exceeds the required range, each area is divided equally for further reduction.

[0063] To ensure low test time and high fault coverage for the detection method in this example, it's important to understand that SA0 can only occur in memristors with low resistance. The read operation during array initialization provides the necessary information about the resistance value. To minimize testing costs, SA0 testing can be avoided in low-resistance regions. Similarly, SA1 faults can only occur in high-resistance memristors, so testing regions without high-resistance memristors is also omitted.

[0064] As shown Figure 7As shown in the figure, the entire grid represents the entire memristor crossbar array, and each small grid represents a memristor. Assuming that the total number of memristors in the entire crossbar array is N, when a memristor in the memristor crossbar array is stuck, the array is divided into multiple parts for detection using the regional detection method, replacing the traditional detection scheme. The fault area determined after the final detection is divided into n shaded areas in the figure, and the stuck faults are distributed in them. The number of memristors pruned in each fault area is N1, N2, N3, ..., N respectively. n ,satisfy:

[0065] N1+N2+N3+...+N n =αN

[0066] where N1, N2, N3, ..., N n is the number of memristors in each of the n fault pruning regions, N is the total number of memristors in the entire memristor crossbar array, and α is the maximum pruning coefficient. In the present invention, the recognition tasks for the MNIST dataset and the CIRAF-10 dataset are 0.7-0.74 and 0.4-0.42 respectively, which is Figure 4 The data obtained in .

[0067] When the range of the column containing the faulty memristor in the array is detected, the "half-step voltage method" and "self-compensation method" are used to prune the memristors within the faulty range. The principle is that if either of a pair of memristors is detected to be faulty, the weight represented by that pair of memristors is considered faulty. The faulty memristor is pruned by adjusting the conductance of the normal memristor in the pair to be the same as (or close to) the conductance of the faulty memristor. A long pulse signal exceeding the threshold voltage is applied to the memristor to be pruned, adjusting the resistance of the memristor in this area to a low or high resistance state. At the same time, the memristors in the compensation column are modulated using the same method to make the sum of the output currents of the two columns zero, completing the pruning process.

[0068] The threshold voltage of the AVTM memristor model selected in the present invention is 0.7V. A DC voltage of 0.5V or -0.5V is applied to the memristor to be adjusted from the top and right of the array respectively. t and -V t , so that the resistance value of the selected memristor can be precisely adjusted. Figure 8 As shown, Figure 8 (a) Corresponding to cases 3, 5, and 7 in Table 1, V is applied over the pruning area for a long enough time. t , the voltage -V is applied on the right side for a long enough time t , so that the conductance values of both columns in the selected area are adjusted to G max , after passing through the subtractor, the output is 0, achieving pruning. Figure 8(b) Corresponding to cases 2, 4, and 6 in Table 1, a voltage of -V is applied above the pruning area. t , voltage V is applied on the right side t , the conductance values of both columns in the region are adjusted to G min When the selected memristor is pruned, the other rows or columns of memristors that do not need to be pruned are grounded. Therefore, the voltage across them is only 0V or ±0.5V, which does not reach the threshold voltage of the memristor. Therefore, the resistance of the other memristors remains unchanged.

[0069] After faulty weights are pruned, the overall performance of the model is inevitably affected, so the weights and biases need to be readjusted. Currently, BP training is a commonly used method to restore model accuracy, which can improve network performance to a certain extent. However, the pruning process removes some important weights or neuron connections, which may lead to a decrease in network reasoning ability. Simple BP training may not be able to restore network performance. Pruning a large number of weights may increase the complexity of the network structure. Irregular pruning processes can change the weight distribution of the network, thereby affecting gradient calculations. During retraining, pruned weight connections may cause gradient vanishing or gradient exploding phenomena.

[0070] To address these issues, this paper proposes using a knowledge distillation training method to replace traditional backpropagation (BP) training. Knowledge distillation trains the student model by learning from the outputs of a teacher model, enabling it to more quickly understand the data distribution. The complete model trained before pruning serves as the "teacher model" for knowledge distillation, helping the pruned "student model" learn knowledge and thus recovering performance lost during the pruning process.

[0071] First, the original model is trained using the data. The trained model is backed up, and the weight matrix is pruned according to the location of the faulty memristor, creating a mask. The pruned network is then retrained using knowledge distillation. The purpose of using knowledge distillation is to improve the accuracy of the pruned sparse network by extracting the softmax from the original deep model to set as the soft target for sparse network training. The original trained model infers the data and generates an output with a probability distribution, which serves as the soft label learned by the pruned model. A temperature parameter T is used to adjust the predicted probability distribution:

[0072]

[0073]

[0074] Where T is the temperature parameter used to adjust the predicted soft target. A higher temperature T will make the probability distribution smoother, thereby providing more inter-class information for the student model. iis the model's prediction score for category i, which is determined by the weight W of the last layer and the input V I The calculation results are:

[0075] z i =W·V I ;

[0076] The pruned student model learns by minimizing the difference between the output and the teacher model, introducing a response-based distillation loss function:

[0077]

[0078] Where T is the temperature coefficient of knowledge distillation, and Represent the output probability distribution of the student model and the teacher model respectively.

[0079] Here, the output of the pruned student model is compared with the output of the teacher model, so that the student model can still capture the global information and inter-class relationships of the data after pruning. During distillation training, the teacher model is not updated simultaneously with the student model, so the teacher model no longer participates in the propagation of gradients.

[0080] Like traditional knowledge distillation, this paper also uses a combination of losses between the student model and the hard label, usually using the cross entropy loss function:

[0081]

[0082] Among them, y i represents the true label of the dataset, represents the output probability distribution of the student model.

[0083] The output of the intermediate layer of the teacher model serves as the knowledge to supervise the training of the student model. Its unpruned intermediate layer feature knowledge is a good supplement to the response knowledge. Therefore, the feature distillation loss function is introduced and defined as follows:

[0084]

[0085] in, and are the feature representations of the teacher model and the student model at layer l, respectively, where L is the number of layers to be aligned. To design an on-chip training architecture for multi-loss knowledge distillation to enable real-time training of pruned models, this example derived and calculated the weight gradient changes generated by the three losses, and selected appropriate coefficients through testing for weighted summation.

[0086] The multi-loss distillation training process is implemented based on the back-propagation process. For the classification layer, the softmax function circuit is used to obtain the probability distribution of the teacher model and the student model. In order to minimize the response distillation loss, the gradient descent method is combined and the weight update rule is:

[0087]

[0088] Where η is the learning rate, is the partial derivative of the response distillation loss with respect to w.

[0089] Use the chain rule of Softmax to find the gradient of the student model's predicted score for each class, and finally integrate them together to obtain the weight update rule under the response distillation loss as follows:

[0090]

[0091] Where T is the temperature coefficient of knowledge distillation, and Represent the output probability distribution of the student model and the teacher model respectively.

[0092] For the final classification layer, in addition to using the response distillation loss function, we also need to use the traditional cross-entropy loss function to allow the student model to learn the true label. Refer to the weight update rule derivation formula of the response distillation loss to obtain the weight update rule under the cross-entropy loss:

[0093]

[0094] Among them, y i represents the true label of the dataset, represents the output probability distribution of the student model.

[0095] In the final classification layer, response distillation loss and cross entropy loss jointly guide weight updates. When the error is backpropagated to the previous layer, the introduction of feature distillation loss can help the pruned network better restore accuracy. Similar to the weight update rule in response distillation loss, the weight update rule can be obtained:

[0096]

[0097] in, and are the feature representations of the teacher model and the student model at layer l, V I Represents the input voltage of the lth layer.

[0098] In the process of error back propagation, when the error propagates to the second layer, there is an error obtained by the joint propagation of three losses. We define the total mixed loss function as follows:

[0099] L total =αL CE +βL Response +γL fea ;

[0100] Among them, α, β and γ are the coefficient factors of three losses, L CE 、L response and L fea They are cross entropy loss, response distillation loss and feature distillation loss respectively.

[0101] To handle detected faults in real time, we designed an on-chip training architecture for multi-loss knowledge distillation based on the derived weight update formula. The weight changes resulting from the three losses are calculated in three modules, and appropriate coefficients are selected through testing for weighted summation. In the online knowledge distillation training circuit architecture, the Softmax function circuit primarily consists of an exponential circuit, a divider, transistors, and an inverse proportional amplifier. The Softmax function circuit obtains the output probability distribution of the student and teacher models for each category of prediction. The input voltage T acts as a temperature coefficient in knowledge distillation; adjusting this voltage can smooth the output probability distribution. Based on the derived weight update formula for the response distillation loss, a response loss calculation module is designed. By comparing the output voltages of the student and teacher models after the Softmax function is activated, the response distillation loss is minimized, allowing the student model to learn the output probability distribution of the teacher model. Furthermore, the student model's output voltage after the Softmax function is compared with the voltage of the true labels in the dataset to minimize the cross-entropy loss, allowing the student model to gain information from the true labels. In addition, the output feature maps of the intermediate layers of the student model and the teacher model are compared to minimize the feature distillation loss between the teacher model and the student model, so that the intermediate layer features of the student model and the teacher model are aligned. The three loss functions jointly supervise the training, so that the pruned student model can be trained more effectively, compensating for the adverse effects caused by large-scale pruning.

[0102] After pruning the faulty memristors, we applied the proposed knowledge distillation technique to repair the memristor neural network with stuck-on faults. Since the knowledge distillation online training circuit architecture uses various loss functions to promote the training of the pruned sparse network, the present invention examines their performance throughout the training process and analyzes their changes during the distillation training process, such as Figure 9 As shown in Figure 2, we can see that the response distillation loss function has the fastest convergence rate, which to some extent limits the guidance of the teacher model to the student model. The feature loss function helps the student model better understand the structure and characteristics of the input data.

[0103] Figure 10 (a) and Figure 10 (b) Results of retraining the LeNet-5 and AlexNet models using knowledge distillation. The results show that after applying knowledge distillation training to the pruned results, the recognition accuracy of both models on the datasets has significantly improved. When the number of faults reaches 10%, the recognition accuracy on the MNIST dataset still reaches 95.28% and 94.5% under the two fault conditions, with 70.71% of the weight parameters pruned. The recognition accuracy on the CIFAR-10 dataset still reaches 70.04% and 68.3%, with 32.81% of the weight parameters pruned.

[0104] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. A circuit for implementing distillation online training of neural network fault tolerance, characterized in that: It includes a student model neural network model circuit, a teacher model neural network model circuit and a knowledge distillation training loss calculation module, and the student model neural network model circuit and the teacher model neural network model circuit are symmetrically distributed; The neural network model circuits of both the student model and the teacher model include several convolutional layers and several fully connected layers, labeled layer 1, layer 2, ... layer n, where n is a natural number greater than 1; the last fully connected layer is the classification layer, and its output is connected to the Softmax activation function circuit. The outputs of the remaining fully connected layers and all convolutional layers are connected to the ReLU activation function circuit. The output of each activated layer constitutes the output feature map; The knowledge distillation training loss calculation module includes the feature distillation loss technology module, the response distillation loss calculation module, and the cross entropy loss calculation module; The output feature map of each layer in the student model is compared with the output feature map of the corresponding layer of the teacher model in the form of voltage, and the student model is aligned with the features of the intermediate layer by minimizing the feature distillation loss calculation module; in the classification layer, the probability distribution of the student model output is compared with the output probability distribution of the teacher model, and the student model learns the output of the teacher model by minimizing the response distillation loss calculation module; similarly, in the classification layer, the output probability distribution of the student model is compared with the true label at the same time, and the student model learns the true label by minimizing the cross entropy loss; the three loss functions jointly supervise the student model for online training, so that knowledge distillation can run efficiently on edge devices.

2. The distillation online training circuit for realizing the fault tolerance function of a neural network according to claim 1, characterized in that: The output feature map of each layer in the student model and the output feature map of the corresponding layer in the teacher model are connected to the feature distillation loss calculation module, and are calculated and compared through the feature distillation loss calculation module.

3. The distillation online training method for realizing the fault tolerance function of a neural network according to claim 1 or 2, characterized in that: The steps include: (1) Apply a voltage exceeding the threshold to all memristors and initialize the resistance to R min , then divide the complete memristor crossbar array into at least 2 regions evenly, and write the same change △R to all memristors; (2) Apply a test voltage to each test area from the input port and calculate the ideal output result as a reference voltage; A shared current sensor with a TIA structure is set at the column port of the memristor crossbar array, and the output result is compared with the reference voltage. If the comparison result has no error, it means there is no fault. When the result is in error, it indicates that there is a stuck fault in the detection area; If the determined range exceeds the maximum limit, the divided area is further divided and further tested using the above method to narrow the fault area. The final fault area is determined and the memristors in the area are marked. (3) The array is retrained using the designed multi-loss knowledge distillation training architecture, and the back-propagation gradient of the weights is calculated. According to the previously marked memristors, if they are marked memristors, the self-compensation method is used for pruning. If they are unmarked memristors, normal updates are performed according to the gradient calculation formula to complete the retraining.

4. The distillation online training method for realizing the fault tolerance function of a neural network according to claim 3, characterized in that: When the memristors in the memristive neural network are fault-free, the LeNet-5 model and AlexNet model constructed using the memristive crossbar array can achieve accuracies of 98.67% and 76.29% on the MNIST and CIFAR-10 datasets respectively; when the memristors are defective, the inference accuracy of the two neural network models constructed using the memristive crossbar array on the two datasets is impaired and cannot meet the accuracy requirements.

5. The distillation online training method for realizing the fault tolerance function of a neural network according to claim 3, characterized in that: By dividing the memristor crossbar array into at least two regions, fault detection is performed on each region instead of the traditional single memristor detection. The same change △R is written to all memristors, and then a test voltage V is applied from the input port to each test region. test , calculate and output the test result △R×V test ; Using the designed fault detection circuit, the output result is compared with the set reference voltage through the static voltage comparison method. If there is a difference with the ideal situation, it is determined to be a fault, otherwise it is considered to be no fault.

6. The distillation online training method for realizing the fault tolerance function of a neural network according to claim 3, characterized in that: The teacher model neural network model circuit and the student model neural network model circuit based on the memristor crossbar array use the 2M method to represent signed weights, and the signed weights are represented by the difference in the conductance of a pair of memristors. The normal memristor in a pair of memristors is used to compensate for the faulty memristor, and the faulty weights are pruned by adjusting the conductance of the normal memristor in the memristor pair to be the same as or close to the conductance of the faulty memristor.

7. The distillation online training method for realizing the fault tolerance function of a neural network according to claim 6, characterized in that: Since a weight is represented by the difference in conductance between a pair of memristors, the situation is different when one or both of the memristors fail. For different situations, the adjustable range of conductance is different, and the weight mapping range also changes. Compensation for the faulty memristor is performed based on the mapping range.

8. The distillation online training method for realizing the fault tolerance function of a neural network according to claim 3, characterized in that: The pruned network architecture is retrained using the knowledge distillation training loss calculation module. The online training architecture can be performed directly on edge devices without relying on offline devices.

9. The distillation online training method for realizing the fault tolerance function of a neural network according to claim 3, characterized in that: Based on the back-propagation principle, online training of multi-loss knowledge distillation of the pruned student model is implemented on the circuit architecture; the output of the pruned student model neural network model circuit is compared with the output of the teacher model neural network model circuit, so that the student model can still capture the global information and inter-class relationship of the data after pruning, and the output of the middle layer of the teacher model neural network model circuit is used as the knowledge to supervise the training of the student model neural network model circuit. The unpruned middle layer feature knowledge is complementary to the response knowledge.

Images