Convolution operator capable of accommodating random bit flipping error and application method thereof

Abstract

The invention discloses a convolution operator capable of containing random bit flipping errors and an application method of the convolution operator. The operator is constructed based on a Winograd algorithm, a completely decoupled calculation path design is adopted, a convolution kernel is directly trained and stored in a Winograd domain, and the influence of a random bit flipping error is limited to a local output region; a sparse optimization operation is introduced, the sparsity of the input feature map is reserved, and the calculation overhead is reduced; integrating a self-reference suppression strategy, and dynamically detecting and suppressing abnormal activation based on the local similarity of a Winograd domain activation value; during application, the fault-tolerant convolution operator can replace traditional convolution and is embedded into an existing neural network structure, and the fault-tolerant capability is improved and the calculation overhead is reduced while normal reasoning performance is not lost; the method has good compatibility and combinability, can be widely embedded into an existing neural network model, effectively improves the fault-tolerant capability of the network model, and is suitable for multi-scene and multi-requirement neural network fault-tolerant design.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence and fault-tolerant computing, and particularly relates to a convolution operator capable of tolerating random bit flip errors and an application method thereof. BACKGROUND

[0002] With the rapid development of deep learning technology, deep neural networks (DNN) have been widely applied to safety-critical scenarios such as autonomous driving, aerospace, medical diagnosis, etc. In actual deployment, these applications often need to run in complex environments such as edge devices, unmanned systems or space electronic platforms, which are prone to random bit flip errors in hardware platforms, which may cause neural network prediction errors or even system crashes in severe cases.

[0003] Random bit flip error is a type of non-permanent hardware failure, usually caused by radiation particle bombardment, voltage fluctuation, aging effect, etc., which causes a certain data bit to flip from 0 to 1 or from 1 to 0 during storage or calculation. When this error occurs in the key calculation path (such as convolution operation) of the neural network, it may cause forward inference errors and quickly spread, destroying the entire feature map structure, affecting the final performance of the model, and even causing catastrophic consequences.

[0004] Current neural network fault-tolerant design mostly adopts methods with high cost and complex processing such as redundant computation (such as multi-version model inference), error detection coding, fault injection training, etc., without fully considering the influence of neural network core operators on fault-tolerant capability. As the core component of feature extraction, the calculation logic of the convolution operator directly determines the fault propagation path and is one of the key factors that determine the fault-tolerant performance. The existing convolution operators (such as standard convolution, conventional Winograd convolution, etc.) lack fault-tolerant capability, which makes bit flip errors easily spread to the entire feature map. Therefore, designing a convolution operator with high fault-tolerant capability and low computational overhead has become an urgent need to improve the reliability of neural networks. SUMMARY

[0005] The application aims to provide a convolution operator capable of tolerating random bit flip errors and an application method thereof, which has strong fault-tolerant capability, low computational overhead and good compatibility. As a basic component of neural networks, the convolution operator can improve the reliability of neural network models in complex environments through completely decoupled calculation path design and self-reference error suppression mechanism, and meet the fault-tolerant deployment requirements of neural networks in multiple scenarios and multiple requirements.

[0006] The application provides a convolution operator capable of tolerating random bit flip errors, which is constructed based on the Winograd algorithm and includes a Winograd transformation module, a sparse optimization module, a weight decoupling calculation module, a self-reference suppression module and a Winograd inverse transformation module.

[0007] Winograd transformation module: performing Winograd linear mapping on the spatial domain input feature map through a preset transformation matrix to generate Winograd domain input data;

[0008] Sparse optimization module: introducing sparse optimization operation after Winograd transformation to preserve the sparsity of the input feature map and reduce subsequent calculation overhead;

[0009] Weight decoupling calculation module: storing and applying convolution kernel weights in the Winograd domain to perform Hadamard product operation on Winograd input data, wherein the convolution kernel is directly trained and stored in the Winograd domain without conversion from the spatial domain, forming a completely decoupled calculation path, thereby limiting the influence of random bit flip errors to a local output area and avoiding the problem of fault propagation to the entire feature map in traditional convolution operation, and improving the fault tolerance performance of the neural network;

[0010] Self-reference suppression module: dynamically detecting and suppressing abnormal activations based on the local numerical similarity of activation values in the Winograd domain to further reduce the risk of error propagation;

[0011] Winograd inverse transformation module: restoring Winograd domain output to spatial domain output feature map through a preset inverse transformation matrix.

[0012] In application, the fault-tolerant convolution operator can be used as a basic component of a neural network, replacing traditional convolution operation, and embedded in existing neural network models without additional structural modifications, improving the fault tolerance of the model and reducing the calculation overhead without loss of normal inference performance.

[0013] In the present application, the Winograd transformation module performs linear transformation on the input feature map through a preset transformation matrix to map the input feature map in the spatial domain to the Winograd domain; specifically including the following steps:

[0014] (1) Feature map division: according to the size of the input feature map C in ×H×W(C in is the number of input channels, H is the height, and W is the width) and the size R×R of the Winograd convolution kernel, the input feature map is divided into several R×R local partitions;

[0015] (2) Load transformation matrix: using Cook-Toom algorithm or based on Lagrange interpolation method to determine the Winograd transformation matrix of the input feature map; for commonly used Winograd convolution configuration, the transformation matrix is known, for example, for 4×4 Winograd convolution kernel, the corresponding input feature map transformation matrix B is:

[0016]

[0017] (3) Perform input feature map Winograd transformation: for each local partition d of the input feature map, perform matrix multiplication operation B T ·d·B, convert the spatial domain feature to the feature representation of the Winograd domain, generate Winograd domain input data with size R x R x C in ;

[0018] (4) Output transformation result: splice the Winograd domain feature map blocks of all local partitions to generate complete Winograd domain input data and serve as the input of the subsequent sparse optimization module.

[0019] In the present application, the sparse optimization module performs a nonlinear activation operation on the Winograd domain input feature map after the Winograd transformation is completed, and the nonlinear activation function is a ReLU (Rectified Linear Unit) function, which sets all negative elements to zero and retains positive elements, so as to maintain the key information representation capability of the input feature map while enhancing the sparsity of the feature and reducing the subsequent calculation overhead.

[0020] In the present application, the weight decoupling calculation module directly trains, stores and applies the convolution kernel weight in the Winograd domain, and performs element-wise multiplication operation on the Winograd input data processed by the sparse optimization module, which specifically includes the following steps:

[0021] (1) Convolution kernel parameter decoupling: initialize and train the convolution kernel weight in the Winograd domain to form a completely decoupled parameter representation form, and store it in the form of a tensor;

[0022] (2) Perform Hadamard product operation: multiply the Winograd domain convolution kernel weight and the Winograd domain input data processed by the sparse optimization module element by element to obtain the intermediate result , wherein W is the Winograd domain convolution kernel weight, and is the Hadamard product operation;

[0023] (3) Output intermediate calculation result: transmit the Hadamard product result to the self-reference suppression module for abnormal activation detection and suppression processing;

[0024] In the above manner, the weight decoupling calculation module can realize complete decoupling of the convolution kernel weight in the calculation path, effectively avoid the propagation risk of random bit flip errors in the convolution kernel, and limit the error to the local output area.

[0025] In the application, the self-reference inhibition module dynamically detects and inhibits abnormal activation values based on the local numerical similarity of Winograd domain activation values; specifically, for each activation value in the Winograd domain Hadamard product result, the activation values in its neighborhood are extracted, the sum of the absolute values of the activation values in the neighborhood is calculated, and the threshold value is multiplied by the threshold coefficient as the inhibition threshold value of the activation value; if the absolute value of the activation value is greater than its corresponding inhibition threshold value, it is determined that the position activation value is an abnormal activation, and inhibition is performed in a manner including but not limited to zeroing, truncation or replacing the inhibition threshold value; the module realizes fault-tolerant judgment without external redundant information by using an internal data-driven adaptive judgment mechanism, can effectively inhibit abnormal activation values caused by bit flip errors, avoid interference with subsequent calculations, and enhance the fault tolerance of the operator.

[0026] In the application, the Winograd inverse transformation module performs a linear inverse transformation operation on the Hadamard product result processed by the self-reference inhibition module through a preset output inverse transformation matrix A to obtain the output feature map in the spatial domain

[0027] In the application, the application method of the convolution operator tolerant to random bit flip errors includes the following steps:

[0028] (1) determining the convolution layer to be replaced in the target neural network, including but not limited to a standard convolution layer or a conventional Winograd convolution layer;

[0029] (2) embedding the convolution operator tolerant to random bit flip errors into the corresponding position of the target neural network to replace the original convolution operator, and maintaining the input channel number, output channel number and feature map size consistent with the original network during embedding;

[0030] (3) training the neural network embedded with the convolution operator tolerant to random bit flip errors using image classification, semantic segmentation, super-resolution and other task data sets;

[0031] (4) in the inference stage, when a random bit flip error occurs, relying on the weight decoupling calculation and self-reference inhibition in the fault-tolerant convolution operator, fault-tolerant inference output is realized to ensure the stable output capability of the neural network in the error environment.

[0032] Compared with the existing neural network operator fault-tolerant enhancement technology, the application has the following advantages:

[0033] First, high reliability, the application realizes effective limitation of the influence range of random bit flip errors through completely decoupled calculation path design; at the same time, combined with the self-reference error inhibition mechanism, the error propagation is further blocked, and the ability of the neural network operator to tolerate random bit flip errors is significantly improved;

[0034] Second, low overhead, the present application inherits the low computational complexity advantage of Winograd algorithm, through sparse optimization module in Winograd transformation after introducing nonlinear activation function operation, preserve feature sparsity to reduce subsequent computing overhead; At the same time, using self-reference error suppression mechanism, without external redundant information can be used for error judgment and suppression, improve fault tolerance ability while avoiding area and power consumption overhead on hardware deployment.

[0035] Third, strong compatibility, the present application as a neural network basic component, can directly replace traditional convolution operator operation, flexible embedded in existing neural network model without additional modification structure, adapt to a variety of tasks;And its fault tolerance mechanism does not depend on specific network type or hardware platform, can compatible with mainstream deep learning framework training and inference process, has wide applicability. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A convolution operator tolerant to random bit flip errors is described.

[0037] Figure 2 A Winograd domain activation value statistical analysis diagram.

[0038] Figure 3 A self-reference suppression schematic diagram.

[0039] Figure 4 A performance comparison diagram before and after applying the fault-tolerant operator of the present application to ResNet-18 network.

[0040] Figure 5 A performance comparison diagram before and after applying the fault-tolerant operator of the present application to VGG-11 network. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the present application clearer and more apparent, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0042] Example 1

[0043] A convolution operator tolerant to random bit flip errors and its application method, as shown in Figure 1 , including Winograd transformation module, sparse optimization module, weight decoupling calculation module, self-reference suppression module and Winograd inverse transformation module.

[0044] In this embodiment, a fault-tolerant convolution operator is built based on the Python language and the PyTorch framework, and is verified in the CIFAR-10 image classification task. The selected application neural network is ResNet-18 and VGG-11, and the 3*3 standard convolution operator in the original network is replaced with the 4*4 size fault-tolerant convolution operator in the application; the hardware platform is a server equipped with NVIDIA GeForce RTX 4090 GPU.

[0045] In this embodiment, the Winograd transformation module specifically includes the following operations:

[0046] (1) Using the PyTorch framework, the input feature map is divided into several 4*4 local partitions, each of which contains local features of the input feature map;

[0047] (2) According to the Winograd algorithm theory, load the transformation matrix B. For the Winograd convolution kernel with a size of 4*4 in this embodiment, the input feature map transformation matrix B is:

[0048]

[0049] (3) Based on the transformation matrix and the divided input feature map local partitions, perform Winograd transformation operation; for each local partition d of the input feature map, perform matrix multiplication B T ·d·B, convert the spatial domain feature to the feature representation of the Winograd domain to generate 4*4 Winograd domain input data

[0050] (4) Concatenate the Winograd domain features of all local partitions to form complete Winograd domain input data as the input of the subsequent sparse optimization module.

[0051] In this embodiment, the sparse optimization module, after the Winograd transformation is completed, performs a nonlinear activation operation on the Winograd domain input feature map, and the nonlinear activation function is the ReLU (Rectified Linear Unit) function, which retains positive value elements and sets negative value elements to zero, so as to maintain the key information representation ability of the feature map while enhancing the sparsity of the feature and reducing the subsequent calculation overhead; at the same time, since the multiplication of zero value elements with any element results in 0, it is also conducive to suppressing random bit flip errors in the weights to some extent and enhancing the fault tolerance of the operator.

[0052] In this embodiment, the weight decoupling calculation module directly trains, stores and applies the convolution kernel weight in the Winograd domain, and performs element-wise multiplication operation on the sparse optimized Winograd input data, specifically including the following steps:

[0053] (1)Convolution kernel parameter decoupling: directly initialize and store the 4x4 Winograd domain convolution kernel W in the Winograd domain, form a fully decoupled parameter representation, without conversion from the spatial domain 3x3 convolution kernel; directly update the 4x4 Winograd domain convolution kernel parameters in the training stage to adapt to the calculation logic of the fault-tolerant convolution operator through back propagation;

[0054] (2) Perform Hadamard product operation: element-wise multiply the Winograd domain 4x4 convolution kernel weight and the Winograd domain input data output by the sparse optimization module to obtain the intermediate result Where W is the Winograd domain convolution kernel weight, and is the Hadamard product operation.

[0055] (3) Output intermediate calculation result: transmit the Hadamard product result to the self-reference suppression module for abnormal activation detection and suppression processing;

[0056] In the above manner, the weight decoupling calculation module can realize complete decoupling of the convolution kernel weight in the calculation path, effectively avoid the propagation risk of random bit flip errors in the convolution kernel, and limit the error to the local output area.

[0057] In this embodiment, the self-reference suppression module dynamically detects and suppresses abnormal activation values based on the local numerical similarity of the Winograd domain activation values, and the specific operation is as follows:

[0058] (1) For each element A ij in the Winograd domain activation, consider its 3x3 neighborhood S = {i-1, i, i+1}, T = {j-1, j, j+1}, and calculate the sum of the absolute values of all elements in the neighborhood s∈S ∑ t∈T |A s,t |;

[0059] (2) Set threshold coefficient γ: In this embodiment, the neighborhood numerical relationship in the Winograd domain activation is counted, and the activation-neighborhood sum ratio As shown in Figure 2 , more than 98% of the Winograd activations satisfy That is, under the error-free state, the Winograd activation values show local similarity; therefore, in this embodiment, the threshold coefficient γ = 2 is set, and the suppression threshold 2∑ s∈S ∑ t∈T |A s,t | is obtained, which suppresses abnormal activations while retaining normal activations;

[0060] (3) Suppression operation: as shown inFigure 3 As shown, if the absolute value of the activation value element exceeds its corresponding inhibition threshold, the position activation value is determined to be an abnormal activation, and is inhibited in a manner including but not limited to zeroing, truncation, or replacement with the inhibition threshold; in this embodiment, the form of replacement with the inhibition threshold is adopted, and when |A i,j |>2∑ s∈S ∑ t∈T |A s,t |,the activation value is replaced with 2∑ s∈S ∑ t∈T |A s,t |,otherwise the original value remains unchanged; that is, the result of the inhibition operation is Wherein sign() is a sign function, and min() is a minimum value function.

[0061] (4) The self-reference inhibition module realizes fault-tolerant judgment without external redundant information by using an internal data-driven adaptive judgment mechanism, can effectively inhibit abnormal activation values caused by bit flip errors, avoid interference on subsequent calculations, and enhance the fault tolerance of the operator.

[0062] In this embodiment, the Winograd inverse transformation module performs a linear inverse transformation operation on the Hadamard product result processed by the self-reference inhibition module by using a preset output inverse transformation matrix A, to obtain the output feature map in the spatial domain The specific steps are as follows:

[0063] (1) Load the inverse transformation matrix A. For the 4x4 Winograd convolution kernel in this embodiment, the inverse transformation matrix A is

[0064]

[0065] (2) Perform a linear inverse transformation operation on the Hadamard product result processed by the self-reference inhibition module to obtain a 2x2 spatial domain output;

[0066] (3) Concatenate all 2x2 spatial domain outputs according to the original positions to generate a complete spatial domain output result.

[0067] In this embodiment, the convolution operator tolerant to random bit flip errors is applied and verified in the CIFAR-10 image classification task. The selected application neural network is ResNet-18 and VGG-11, the 3x3 standard convolution operator in the original neural network is replaced with the 4x4 size fault-tolerant convolution operator in this embodiment, and the input channel number, output channel number and feature map size are kept consistent with the original network; the neural network embedded with the fault-tolerant convolution operator is trained using the CIFAR-10 dataset, and inference and error injection experiments are performed; as shown in Figure 4 ,Figure 5 As shown, the accuracy of the neural network embedded with the fault-tolerant convolution operator is the same as that of the original network in the fault-free scenario; when the error injection rate is 5x10 -7 When the random bit flip error is injected at a rate of 5x10-4, the accuracy of the ResNet-18 network and the VGG-11 network using the standard convolution is 16% and 30% respectively, while the accuracy of the neural network embedded with the fault-tolerant algorithm is 84% and 86% respectively, and the fault-tolerant capability is significantly improved, and the computational overhead is reduced by 22% and 18% (measured in FLOPs) respectively. The present application has good compatibility, fault tolerance and low overhead, can be widely embedded in existing neural network structures, effectively improves the fault tolerance of the neural network model, and is suitable for neural network fault tolerance design in multiple scenes and multiple requirements.

[0068] The above examples are only for illustrating the technical concept and characteristics of the present application, and the purpose is to more clearly set forth the object, technical scheme and advantages of the present application, and are not used to limit the protection scope of the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A convolution operator tolerant to random bit flipping errors, which, when applied, serves as a fundamental component of a neural network, replacing traditional convolution operations and being embedded in existing neural network models, characterized in that... It is designed based on the Winograd algorithm and includes a Winograd transformation module, a sparse optimization module, a weight decoupling calculation module, a self-reference suppression module, and a Winograd inverse transformation module; among which: Winograd transformation module: Performs Winograd linear mapping on the spatial domain input feature map using a preset transformation matrix to generate Winograd domain input data; Sparse optimization module: After the Winograd transformation, a sparse optimization operation is introduced to preserve the sparsity of the input feature map and reduce the subsequent computational overhead. Weight decoupling calculation module: Stores and applies convolution kernel weights in the Winograd domain, performs Hadamard product operation on Winograd input data, wherein the convolution kernel is directly trained and stored in the Winograd domain to achieve complete decoupling of convolution kernel weights in the calculation path, and limits the impact of random bit flipping errors to the local output region. Self-reference suppression module: Based on the local numerical similarity of activation values ​​in the Winograd domain, it dynamically detects and suppresses aberrant activations, reducing the risk of error propagation; Winograd inverse transform module: Using a preset output inverse transform matrix, it performs a linear inverse transform operation on the Hadamard product result after processing by the self-reference suppression module, restoring the Winograd domain output to the spatial domain output feature map.

2. The convolution operator tolerant to random bit flipping errors according to claim 1, characterized in that, The Winograd transformation module performs a linear transformation on the input feature map using a preset transformation matrix, mapping the input feature map from the spatial domain to the Winograd domain; specifically, it includes the following steps: (1) Feature map partitioning: Based on the size C of the input feature map in ×H×W and Winograd convolution kernel size R×R, C in H is the number of input channels, H is the height, and W is the width. The input feature map is divided into several R×R local partitions. (2) Loading the transformation matrix: The Cook-Toom algorithm or the Lagrange interpolation method is used to determine the Winograd transformation matrix of the input feature map; (3) Perform Winograd transformation on the input feature map: Perform matrix multiplication B on each local partition d of the input feature map. T ·d·B transforms the spatial domain features into Winograd domain feature representations, generating a representation of size R×R×C. in Input data for the Winograd domain; (4) Output transformation results: stitch together the Winograd domain feature maps of all local partitions to generate complete Winograd domain input data, and use it as input to the sparse optimization module.

3. The convolution operator tolerant to random bit flipping errors according to claim 1, characterized in that, The sparse optimization module performs a nonlinear activation operation on the Winograd domain input feature map after the Winograd transformation is completed. The nonlinear activation function is the ReLU function.

4. The convolution operator tolerant to random bit flipping errors according to claim 1, characterized in that, The weight decoupling calculation module directly trains, stores, and applies convolutional kernel weights in the Winograd domain, and performs element-wise multiplication on the Winograd input data processed by the sparse optimization module. Specifically, it includes the following steps: (1) Decoupling of convolution kernel parameters: The convolution kernel weights are initialized and trained in the Winograd domain to form a completely decoupled parameter representation, which is then stored in tensor form; (2) Perform Hadamard product operation: Perform element-wise multiplication of the Winograd domain convolution kernel weights with the Winograd domain input data processed by the sparse optimization module to obtain intermediate calculation results. Where W is the weight of the Winograd domain convolution kernel, and ⊙ is the Hadamard product operation; (3) Output intermediate calculation results: output the Hadamard product results The data is then transmitted to the self-reference suppression module for abnormal activation detection and suppression.

5. The convolution operator tolerant to random bit flipping errors according to claim 1, characterized in that, The self-reference suppression module dynamically detects and suppresses abnormal activation values ​​based on the local numerical similarity of activation values ​​in the Winograd domain. Specifically, for each activation value in the Hadamard product result of the Winograd domain, the activation values ​​in its neighborhood are extracted, the sum of the absolute values ​​of the activation values ​​in the neighborhood is calculated, and multiplied by a threshold coefficient as the suppression threshold for the activation value. If the absolute value of the activation value is greater than its corresponding suppression threshold, the activation value at that position is determined to be abnormal activation, and suppression is carried out by any one of the following methods: setting it to zero, truncating it, or replacing it with the suppression threshold.

6. A method for applying the convolution operator tolerant to random bit flipping errors as described in claim 1, characterized in that, Includes the following steps: (1) Determine the convolutional layers that need to be replaced in the target neural network, including standard convolutional layers or regular Winograd convolutional layers; (2) The convolution operator that is tolerant to random bit flipping errors is embedded into the corresponding position of the target neural network to replace the original convolution operator. During the embedding process, the number of input channels, the number of output channels, and the feature map size are kept consistent with the original network. (3) Train a neural network embedding the random bit-flipping error-tolerant convolution operator using the target task dataset; the target tasks include image classification, semantic segmentation and super-resolution tasks. (4) During the inference stage, when a random bit flip error occurs, the fault-tolerant inference output is achieved by relying on the weight decoupling calculation and self-reference suppression features in the fault-tolerant convolution operator, thus ensuring the stable inference capability of the neural network under error conditions.

Images