A Fault Tolerance Enhancement Method for Neural Network Models Based on Fine-Grained Error Blocking

Abstract

The present invention discloses a method for enhancing the fault tolerance of a neural network model based on fine-grained error blocking, which is applied to the field of network security. In view of the problem that existing neural network error blocking methods all seek thresholds in layers, and since the thresholds of each channel in a layer are not the same, the thresholds found may be somewhat different from their optimal solutions, the present invention first injects single-bit errors and multi-bit errors into a deployed neural network model, and saves the original weights and the weight model with errors as a weight data set; then, based on the weight data set, a genetic algorithm is used to perform multiple iterations to obtain the thresholds of each channel in each layer of the neural network; finally, a custom activation function is generated based on the obtained thresholds, and the activation function is replaced in the target model, thereby achieving enhanced fault tolerance of the neural network.

Description

Technical Field

[0001] The present invention belongs to the field of network security, and in particular relates to a neural network model fault-tolerant technology for image classification. Background Art

[0002] With the rapid development of deep neural networks (DNNs), they are widely used in various fields, such as computer vision applications. These programs mainly complete the task of classifying input samples. That is, for an input image, after inference by a neural network model, it can output the category to which it belongs in a very short time.

[0003] However, considering the complex external environment and highly integrated hardware, the reliability of neural networks is difficult to guarantee, which can lead to immeasurable consequences for the entire application.

[0004] Traditional fault-tolerance enhancement techniques, such as adding redundancy, modifying training algorithms, and optimizing neural networks, rarely consider the characteristics of DNNs, resulting in high overhead and low performance, making them impractical in large-scale computing systems. Hardware-level fault-tolerance enhancement techniques often require expensive hardware redundancy or additional space overhead, which also results in extremely high costs.

[0005] Based on the characteristics of DNNs, existing researchers have discovered that errors occurring in the highest bit of the exponential domain can cause significant perturbations, affecting the final output, and that these significant errors always propagate to every feature map. This means that error propagation can be suppressed by limiting the values of feature maps, thereby achieving enhanced fault tolerance. This is called error blocking.

[0006] Activation layers are a crucial component of DNNs, often following convolutional layers to enhance the neural network's nonlinear mapping. Different activation functions have distinct characteristics. Some, such as tanh, have bounded outputs, while others, such as ReLU, have unbounded outputs. These unbounded activation functions are unable to constrain their outputs when faults occur, ultimately leading to failure. Therefore, adding bounds to these activation functions can help enhance the fault tolerance of DNNs. Furthermore, since activation layers essentially map inputs to outputs, only minor changes to the activation function are needed to modify the output. Therefore, most error clipping techniques are implemented in activation layers.

[0007] Error clipping is a two-step process. The first step is to determine the output threshold. It's important to note that increasing the activation function bounds will inevitably affect the DNN's raw accuracy, so the choice of threshold depends largely on the acceptable loss of raw accuracy. The second step is to insert the threshold into the activation function. After completing these two steps, the DNN has achieved error clipping, and the output of each layer of the neural network is now constrained by the activation function bounds during operation.

[0008] Existing error blocking methods all search for thresholds on a layer-by-layer basis. However, since the thresholds for each channel in a layer are different, the thresholds found may differ from the optimal solution. At the same time, the number of layers in existing neural networks continues to increase. To prevent problems such as vanishing gradients caused by the increase in the number of layers, more and more networks are beginning to use residual blocks to avoid this problem. However, the use of residual blocks can cause errors to propagate secondary within the neural network and further widen the gap in thresholds between different channels in the same layer, thereby reducing the fault tolerance of existing error blocking methods. Most current neural networks use residual blocks in their structures, so new methods are needed to ensure the fault tolerance of neural networks. Summary of the Invention

[0009] To solve the above technical problems, the present invention proposes a fault-tolerant enhancement method for neural network models based on fine-grained error blocking. A weight data set consisting of original weights and weights with errors is used, and a genetic algorithm is used to perform multiple iterations to obtain a suitable threshold for each channel, which is then injected into the activation function to implement a fault-tolerant enhancement method with fine-grained error blocking.

[0010] The technical solution adopted by the present invention is: a method for enhancing the fault tolerance of a neural network model based on fine-grained error blocking, comprising:

[0011] S1. Read the weight file of the target neural network in the computer vision application and save the weight file directly as the first type of weight, i.e., the original weight;

[0012] S2. Inject a single-bit error into each convolutional layer of the target neural network weights. The specific steps are as follows:

[0013] S21: Input the classification image, identify the input image through the target neural network model, calculate the gradient layer by layer based on the recognition result, and select the weight position with the largest gradient in each layer as the location where the bit error occurred;

[0014] S22: Convert the weight to be injected with error from decimal to IEEE754 standard format;

[0015] S23: Select the highest bit of the exponent field and perform bit flipping;

[0016] S24: Convert the flipped error weight to decimal;

[0017] S25: Inject the error weight into the corresponding layer;

[0018] S26: Save the weight with the single-bit error as the second type of weight;

[0019] S3. Calculate the number of weights that need to be bit-flipped according to the set error rate, randomly select the error location and the specific flipped bit, and inject multi-bit errors into the target neural network; save the weights with multi-bit errors as the second type of weights;

[0020] S4, integrating the three types of weights obtained in steps S1, S2, and S3 as the final weight data set;

[0021] S5. Based on the obtained weight data set, a genetic algorithm is used to perform multiple iterations, and a threshold value of each channel is determined according to the iteration results and the set parameters;

[0022] S6. Generate a corresponding custom activation function based on the obtained threshold, replace it with the corresponding neural network and run the model;

[0023] S7. During operation, whether a serious error occurs is determined based on whether the output exceeds the threshold of the corresponding channel, and serious errors are blocked to achieve the purpose of fault tolerance enhancement.

[0024] Beneficial effects of the present invention: Based on the propagation characteristics of neural networks, the method of the present invention uses a genetic algorithm to determine the threshold of each channel and inserts the threshold into the corresponding activation function to enhance the fault tolerance of the neural network. Compared with traditional methods, the method of the present invention can provide higher fault tolerance, especially for deep neural network models containing residual blocks. In addition, the method of the present invention does not cause excessive additional overhead during operation. The method of the present invention has the following advantages:

[0025] 1. The method of the present invention uses a fine-grained error truncation strategy, expanding the threshold from the coarse-grained standard based on layers in existing work to the fine-grained standard based on channels. This can effectively avoid the problem of insufficient fault tolerance of existing methods caused by differences in theoretical optimal thresholds between channels in deep neural networks with structures such as residual blocks.

[0026] 2. Using a genetic algorithm to iterate multiple times to determine the threshold value can more effectively approach the global optimal solution and avoid being trapped in a local optimal solution. At the same time, users can customize the fitness function to ensure a good balance between the original accuracy loss caused by the fault-tolerant method and the fault-tolerant capability it provides, thereby meeting the versatility in various scenarios.

[0027] 3. In view of the above two advantages, the method of the present invention is suitable for enhancing the fault tolerance of computer vision applications based on neural network models. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of generating a weighted data set according to the present invention.

[0029] Figure 2 This is a flow chart of threshold value determination based on genetic algorithm iteration of the present invention.

[0030] Figure 3 It is a flow chart of the generation of the custom activation function of the present invention.

[0031] Figure 4 This is a flowchart of the neural network fault tolerance enhancement based on fine-grained error blocking of the present invention. DETAILED DESCRIPTION

[0032] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0033] The present invention mainly includes weight dataset generation, threshold determination based on genetic algorithm iteration, custom activation function generation, and fault-tolerance enhancement of deep neural network models based on fine-grained error blocking. The following is an example of fault-tolerance enhancement of a neural network-based image processing program:

[0034] 1. Weight dataset generation primarily involves collecting the original weights and erroneous weights for deployed neural network models. To simulate bit-flip errors generated in hardware, a strictly controlled error injection method is required. Since the target network weights are stored in the application using the IEEE 754 standard, the error injection process requires converting the weights to 32-bit floating point format, performing a bit-flip, and then reconverting to decimal.

[0035] The number of weights in the weight dataset depends on the number of layers in the target network and the user-defined parameters. Assuming a given neural network has N convolutional layers, only one original weight needs to be saved; for weights containing single-bit errors, N weights need to be saved, meaning a single-bit error is injected into each layer. Furthermore, based on Taylor expansion, it can be determined that the impact on the output is related to the gradient of the weights and the specific error:

[0036]

[0037] Among them, |Δy| is the change in output, which can be regarded as the impact of the error on the output. F(w, x) represents the mapping relationship between the weight w used in the current layer and the input x. ε represents the error generated by injecting a single-bit error. Therefore, the specific location where the bit flip occurs can be determined based on the gradient of the weight of each layer. In order to make the impact of the single-bit error large enough, the highest bit in the exponent field of the weight with the highest gradient is usually selected for bit flipping. For multi-bit errors, several weights are saved according to the parameters set by the user. This type of weight is more of a supplement to the second type of error weight.

[0038] The specific process of weighted dataset is as follows Figure 1 As shown, the following steps are included:

[0039] Step A1: Read the weight file of the neural network in the computer vision application and save the weight file directly as the first type of weight, that is, the original weight.

[0040] Step A2: Inject a single-bit error into each convolutional layer of the target neural network weights. The specific steps are as follows:

[0041] Step A2.1: Input the classification image and recognize the input image through the target neural network model. Calculate the gradient layer by layer based on the recognition result, and select the weight position with the largest gradient in each layer as the location where the bit error occurred.

[0042] Step A2.2: Convert the weights to be injected with errors from decimal to IEEE 754 standard format.

[0043] Step A2.3: Select the most significant bit of the exponent field and perform bit flipping.

[0044] Step A2.4: Convert the flipped error weights to decimal.

[0045] Step A2.4: Inject the error weights into the corresponding layers.

[0046] Step A2.5: Save the weights with single-bit errors as the second type of weights.

[0047] Step A3: Calculate the number of weights that need to be bit-flipped according to the set error rate, randomly select the error location and the specific bit to be flipped, and inject multiple bit errors into the target neural network. The error rate is generally set from 1×10 -7 to 1×10 -3 At the same time, in order to reduce time consumption, nine values are usually averaged in one order of magnitude, that is, in 1×10 -7 to 1×10 -6 Take 1×10 -7 , 2×10 -7 , ..., 9 × 10-7 , and so on for other ranges. The specific process of injecting multiple-bit errors can be viewed as the superposition of multiple single-bit error injections (but the highest bit of the exponent field is not necessarily flipped). Therefore, the steps are similar to steps A2.1-A2.5. The difference is that in step A2.1, the weight position with the largest gradient is no longer selected as the error location, but several positions are randomly selected as the bit error location. In step A2.3, the highest bit of the exponent field is no longer selected for bit flipping, but a random bit is selected for flipping.

[0048] Step A4: Integrate the three types of weights obtained in steps A1, A2, and A3 as the final weight data set, which is used to iteratively calculate the threshold of each channel in step 2.

[0049] 2. Threshold selection based on genetic algorithm iteration: To a certain extent, the fault tolerance enhancement method based on error blocking can be regarded as a multi-objective optimization problem, that is, the fault tolerance is as large as possible and the loss of original accuracy is as small as possible. Therefore, for this problem, the present invention adopts genetic algorithm to solve it, which can find the global optimal solution in the search space as much as possible, rather than falling into the local optimal solution; the specific process of genetic algorithm is as follows Figure 2 As shown, the following steps are included:

[0050] Step B1: Design a fitness function based on the optimization goal and use it to evaluate the quality of each solution. The default fitness function for this solution is:

[0051]

[0052] Where Φ represents fitness, G(w, x, t) represents weight w, input x, threshold t, and the output is the final accuracy. When t is missing, the neural network is considered to have no threshold added. This represents the final accuracy achieved by a normal neural network model for the input image x, given the weight w. When the weight is w + ε, this indicates an error ε in the weight. Adding a threshold to the neural network will cause the network model to experience a slight decrease in accuracy when running without errors, but improve accuracy when running with errors due to enhanced fault tolerance. Therefore, G(w + ε, x, t) - G(w + ε, x) can be considered the neural network's fault tolerance after errors, while G(w, x) - G(w, x, t) can be considered the accuracy loss of the neural network when running without errors. ξ is a parameter representing the relative importance of the two parameters. A larger ξ indicates a greater emphasis on the accuracy loss of the neural network when running without errors. Therefore, the maximum fault tolerance is sought while minimizing the loss in accuracy. Conversely, a smaller ξ indicates a greater emphasis on fault tolerance.

[0053] For example, in a given computation, G(w+ε,x) and G(w,x) are 10% and 90%, respectively. There are two thresholds, t1 and t2. The results corresponding to t1, G(w+ε,x,t1) and G(w,x,t1), are 70% and 88%, respectively; while the results corresponding to t2, G(w+ε,x,t2) and G(w,x,t2), are 80% and 80%, respectively. Therefore, when ξ is 0, the fitness values corresponding to t1 and t2 are calculated to be 0.6 and 0.7, respectively, meaning that t2 is superior to t1. In this case, the focus is on the neural network's fault tolerance after errors. When ξ is 10, the fitness values corresponding to t1 and t2 are calculated to be 0.5 and 0.35, respectively, meaning that t1 is superior to t2. In this case, the focus is on the accuracy loss of the neural network when there are no errors. ξ is generally set to 1.

[0054] Step B2: Determine the value range of the coding gene to avoid a large search range that would lead to huge time overhead. It is usually set to [μ×t 0 ,ρ×t 0 ]. Where t 0 The maximum value of each channel in each layer output to the next layer when there are no errors. This is obtained by inputting a classification image, using the target neural network model to identify the input image, and obtaining the maximum output value of each channel in each layer. μ and ρ are two constants used to adjust the search range. Generally, μ is set between 0 and 1, and ρ is set between 1 and 2.

[0055] Step B3: Generate the initial population, and try to ensure that its genes can evenly fill the search space. The specific generation formula is as follows:

[0056]

[0057] Here, i represents the i-th population. There are n populations in the primary population, and n is usually 200.

[0058] Step B4: Calculate the fitness of each group and sort them in descending order.

[0059] Step B5: Determine whether the fitness result meets the requirements or whether the number of iterations has reached a certain number. If so, proceed to step B11; otherwise, proceed to step B6. It is usually set that the fitness reaches 0.65 to meet the requirements, or the iteration can be stopped when the number of iterations reaches 200.

[0060] Step B6: Select the population genes as the parent to directly generate offspring according to the set probability Ω. The probability that the genes of the i-th population in the parent can be directly inherited to the next generation is usually defined as follows:

[0061]

[0062] Obviously, the smaller i is, the greater the probability that the genes of the group can be inherited to the next generation, so that the group with high fitness is more likely to pass on to the next generation than the group with low fitness.

[0063] Step B7: Select the top 20 populations after sorting as the populations with the highest fitness, pair them up as parents, and perform gene crossover at the layer granularity to generate offspring. The specific method is to select a threshold for each layer for crossover. Therefore, after completing the crossover, a group of populations can generate a total of 2N new offspring.

[0064] Step B8: Select the top 20 populations after sorting as the high-fitness population and the bottom 20 populations after sorting as the low-fitness population. Select a pair of these populations as the parent generation, and perform gene crossover at the channel granularity to generate offspring. The specific crossover formula is as follows:

[0065]

[0066] in, and Represent the jth and kth populations in the i-th generation population respectively. Λ is a 0-1 matrix with the same dimension as the population. For a population with high fitness, it is necessary to satisfy Λ j +Λ k =1. In addition, is the Hadamard product of two matrices.

[0067] Step B9: Select a population based on probability to perform gene mutation to generate offspring. The general mutation probability is set to 0.005. The mutation matrix is:

[0068]

[0069] Among them, ∈ is half of the value range of the coding gene, and γ represents the number of discrete variable values in the range [-∈, ∈]. The smaller γ is, the greater the amplitude of the variation is likely to be, while the larger γ is, the smaller the amplitude of the variation is likely to be. Generally, γ is set to 100, and dim(x) ensures that the dimension of the mutation matrix is the same as the dimension of the population. Therefore, the threshold set after mutation can be obtained based on the mutation matrix:

[0070]

[0071] in, It is like Λ j A 0-1 matrix, It still represents the Hadamard product of two matrices.

[0072] Step B10: Take all generated offspring as the population of the new generation and execute step B4.

[0073] Step B11: Select the group output with the highest fitness as the final result.

[0074] 3. Custom activation function generation: After obtaining the thresholds for each channel, it is necessary to make certain changes to the existing activation function based on these data to generate a custom activation function to ensure that the output of each channel can be limited. If you choose to decompose the output of a layer according to the channel, add the corresponding activation function to it for filtering, and then synthesize it into a multi-channel output, it will cause extremely serious additional time overhead. At the same time, the original activation function can accept a number as a threshold to limit the output of the entire layer. Therefore, by modifying some of the underlying code, the activation function can accept a list of thresholds. The length of the list corresponds to the number of channels of the layer, thereby achieving error blocking of the channel. The specific process is as follows Figure 3 As shown, the following steps are included:

[0075] Step C1: Select the activation function that needs to be modified.

[0076] Step C2: Load the threshold values of each channel obtained by the genetic algorithm.

[0077] Step C3: Calculate the number of channels in each layer and cut the threshold into several parts in sequence. Each part contains a threshold corresponding to the number of channels in the layer.

[0078] Step C4: Modify the underlying code so that it can read the threshold value in layers.

[0079] 4. Neural network fault tolerance enhancement based on fine-grained error blocking: After the new neural network is deployed, it mainly uses customized activation functions and thresholds of each channel to limit the output to minimize the impact of serious errors on the output. The specific process is as follows: Figure 4 As shown, the following steps are included:

[0080] Step D1: Based on the custom activation function, determine whether the output of each channel exceeds the corresponding threshold. If it exceeds the threshold, it is considered that the output may be affected by a serious error. If it does not exceed the threshold, it is considered that the output is not affected by a serious error and jumps to step D3.

[0081] Step D2: Limit the excess portion according to the set threshold to ensure that the output of each channel does not exceed the corresponding threshold.

[0082] Step D3: Determine whether it is the last layer. If so, output the result. If not, jump to step D1.

[0083] Those skilled in the art will appreciate that the embodiments described herein are intended to aid the reader in understanding the principles of the present invention, and it should be understood that the scope of the present invention is not limited to such specific descriptions and embodiments. Various modifications and variations are readily apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims.

Claims

1. A method for enhancing the fault tolerance of neural network models based on fine-grained error blocking, characterized in that: include: S1. Read the weight file of the target neural network in the computer vision application and save the weight file directly as the first type of weights; S2, injecting a single-bit error into each convolutional layer of the target neural network weights to obtain weights with single-bit errors as the second type of weights; The specific steps are as follows: S21: Input the classification image, identify the input image through the target neural network model, calculate the gradient layer by layer based on the recognition result, and select the weight position with the largest gradient in each layer as the location where the bit error occurred; S22: Convert the weight to be injected with error from decimal to IEEE754 standard format; S23: Select the highest bit of the exponent field and perform bit flipping; S24: Convert the flipped error weight to decimal; S25: Inject the error weight into the corresponding layer; S26: Save the weight with the single-bit error as the second type of weight; S3. Calculate the number of weights that need to be bit-flipped according to the set error rate, randomly select the error location and the specific bit to be flipped, and inject multi-bit errors into the target neural network; Save the weights with multi-bit errors as the second type of weights; S4, integrating the three types of weights obtained in steps S1, S2, and S3 as the final weight data set; S5. Based on the obtained weight data set, a genetic algorithm is used to perform multiple iterations, and a threshold value of each channel is determined according to the iteration results and the set parameters; S6. Generate a corresponding custom activation function based on the obtained threshold, replace it with the corresponding neural network and run the model; S7. During operation, whether a serious error occurs is determined based on whether the output exceeds the threshold of the corresponding channel, and serious errors are blocked to achieve the purpose of fault tolerance enhancement.

2. A method for enhancing the fault tolerance of a neural network model based on fine-grained error blocking according to claim 1, characterized in that: Step S5 specifically includes the following sub-steps: Step B1: Design fitness function: Where Φ represents fitness, G(w,x,t) represents weight w, input x, threshold t, and the output is the final accuracy; G(w,x) is the final accuracy obtained by a normal neural network model for recognizing the input image x under a given weight w; ξ is the set parameter; Step B2: Determine the value range of the coding gene as [μ×t 0 ,ρ×t 0 ]; where t 0 is the set of maximum values output by each channel of each layer to the next layer when there is no error, μ and ρ are two constants; Step B3: Generate the primary population. The genes of the primary population can evenly fill the search space. The specific generation formula is as follows: Where i represents the i-th group, and there are n groups in the initial generation; Step B4: Calculate the fitness of each group and arrange them in descending order; Step B5: Determine whether the fitness result meets the requirements or whether the number of iterations reaches a certain number. If so, execute step B11; otherwise, execute step B6. Step B6: Select the population genes corresponding to the first M fitness values obtained in step B4 as the parent generation to directly generate offspring according to the set probability; Step B7: Select the populations corresponding to the first M fitness values obtained in step B4 and group them in pairs as the parent generation, and perform gene crossover at the layer granularity to generate offspring; Step B8: Select one population from each of the first M fitness values obtained in step B4 and the last M fitness values obtained in step B4 as the parent generation, and perform gene crossover at the channel granularity to generate offspring. Step B9: Select a population based on probability to perform gene mutation and generate offspring; Step B10: Take all offspring generated in steps B6-B9 as the population of the new generation and execute step B4; Step B11: Select the group output with the highest fitness as the final result.

3. The method for enhancing the fault tolerance of a neural network model based on fine-grained error blocking according to claim 2, characterized in that: In step B2, t 0 The method of obtaining is as follows: input the classification picture, identify the input picture through the target neural network model, and obtain the maximum output value of each channel in each layer.

4. The method for enhancing the fault tolerance of a neural network model based on fine-grained error blocking according to claim 3 is characterized in that: The set probability in step B6 is calculated as follows: Among them, Ω(i) represents the probability that the genes of the i-th population can be directly inherited to the next generation, and n represents the total number of populations.

5. The method for enhancing the fault tolerance of a neural network model based on fine-grained error blocking according to claim 4 is characterized in that: The mutation matrix used for gene mutation in step B9 is: Among them, ∈ is half of the range of the coding gene, γ represents the number of discrete variable values in the range [-∈,∈], and dim(x) ensures that the dimension of the mutation matrix is the same as the dimension of the population.

6. The method for enhancing the fault tolerance of a neural network model based on fine-grained error blocking according to claim 5, characterized in that: In step S6, the corresponding custom activation function is generated according to the obtained threshold value. The specific process is as follows: Step C1: Select the activation function that needs to be modified; Step C2: Loading the threshold values of each channel determined in step S5; Step C3: Calculate the number of channels in each layer and cut the threshold into several parts in sequence. Each part contains a threshold corresponding to the number of channels in the layer. Step C4: Modify the underlying code so that it can read the threshold value in layers.

Images