A residual path-aware FPGA fault-robust quantization-aware training method and system

Abstract

The application discloses a residual path-aware FPGA fault-robust quantization-aware training method and system. The method identifies the main branch path and the shortcut branch path before the residual addition node in the quantization neural network, obtains the path sensitivity through software bit-flipping fault injection, extracts hardware exposure information such as the first-in-first-out queue (FIFO) depth, stream width and storage bit number from the FPGA high-level synthesis data stream, and constructs a path-level risk index composed of the sensitivity, spatial exposure and temporal exposure. According to the risk index, the fault injection probability, the robust loss weight or the quantization protection strength in the quantization-aware training is adjusted, so that the fault robustness of the residual connection quantization neural network when deployed on the FPGA is improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence model compression, quantization-aware training, neural network hardware deployment, and FPGA fault tolerance technology. In particular, it relates to a method and system for performing residual path risk modeling and fault-robust quantization-aware training in an FPGA / HLS dataflow structure for quantization neural networks with residual connections. Background Technology

[0002] With the increasing application of deep neural networks in embedded devices, industrial edge computing, and real-time intelligent sensing scenarios, using FPGAs to accelerate quantized neural networks has become a common solution. FPGAs are characterized by reconfigurability, high parallelism, and high energy efficiency, making them suitable for mapping neural network operators such as convolution, pooling, and element-wise addition into streaming hardware pipelines.

[0003] Quantization-aware training can simulate low-bit-width computation during the training phase, thereby reducing the accuracy loss after the model is deployed to fixed-point hardware. Existing quantization-aware training usually focuses on weight quantization error, activation quantization error, overall model accuracy, and hardware resource constraints, but rarely considers hardware exposure factors such as actual dwell time, FIFO depth, flow width, and path differences in FPGA dataflow implementation.

[0004] Fault-aware training, or fault-injected training, has been used to improve the robustness of neural networks to bit flips, memory perturbations, or computational errors. For example, published research has proposed injecting hardware errors during training to enable quantized neural networks to maintain high inference reliability in the presence of hardware faults.

[0005] The aforementioned techniques can address the issues of fault robustness or hardware-aware quantization in neural networks, but they still have shortcomings: First, fault injection is often handled holistically by layer, operator, or parameter, without modeling the hardware data flow differences between the main branch path and shortcut branch path in the residual network; Second, there is a lack of unified risk indicators between the software sensitivity during the training phase and the actual FIFO depth, flow width, and path dwell time formed after high-level synthesis on the FPGA; Third, for the two paths before residual addition, existing methods struggle to determine which path should receive stronger fault injection constraints or protection strength during training.

[0006] In neural networks with residual connections, the main branch path typically involves computations such as convolution and activation, while shortcut branch paths may involve identity mapping, channel adjustment, or downsampling mapping. In the software model, shortcut branches may appear to have less computation, but in FPGA streaming dataflow implementations, to align with the main branch at the residual addition node, shortcut branches may require deeper FIFO buffers or bandwidth adjustment structures. Therefore, judging path risk solely based on the software model structure is insufficient. Summary of the Invention

[0007] Technical problems to be solved

[0008] This invention aims to address the problem that existing quantitative sensing training and fault sensing training do not fully consider the different fault exposure levels of the main branch path and shortcut branch path in the residual network in the FPGA data flow hardware implementation.

[0009] Specifically, the present invention aims to solve the following problems:

[0010] 1. How to identify the main branch path and shortcut branch path before residual addition in a quantized neural network with residual connections;

[0011] 2. How to measure the sensitivity of different residual paths to software fault injection;

[0012] 3. How to extract path-level hardware exposure information from FPGA high-level synthesis or intermediate dataflow representation;

[0013] 4. How to combine software sensitivity with hardware exposure information into a path-level risk indicator that can be used for training scheduling;

[0014] 5. How to adjust the quantitative perception training process based on the path-level risk indicators to make high-risk paths more robust to faults.

[0015] Technical solution

[0016] To address the aforementioned issues, this invention proposes a residual path-aware FPGA fault robust quantization-aware training method. This method includes: acquiring a quantized neural network model containing residual connections; identifying residual addition nodes and their preceding main branch paths and shortcut branch paths; performing software bit-flip fault injection on the paths to obtain path-level software fault sensitivity; converting the model into a high-level FPGA synthesis dataflow representation, and extracting hardware exposure information such as FIFO depth, flow width, and array parallelism for each path from this dataflow representation; constructing path-level risk indicators; and adjusting the fault injection probability, robust loss weight, or quantization protection strength during training based on the risk indicators.

[0017] The path-level risk indicator in this invention can be expressed as:

[0018]

[0019] in, Indicates a residual layer or residual block. Indicates the path type. Indicates software fault sensitivity. Indicates the amount of space hardware exposed. This indicates the amount of hardware exposure over time.

[0020] Beneficial effects

[0021] Compared with the prior art, the present invention has at least the following beneficial effects:

[0022] 1. It can distinguish the failure risks of the main branch path and the shortcut branch path in the residual connection, instead of processing the entire residual block uniformly;

[0023] 2. It can combine software fault sensitivity with the FIFO depth, current width, and data dwell time formed after FPGA high-level synthesis;

[0024] 3. It can prioritize enhancing the fault robustness of high-risk paths during the training phase, thereby improving the efficiency of training resources and fault injection budget utilization;

[0025] 4. It can provide path-level risk interpretation for quantization models before FPGA deployment, which is beneficial for differentiated quantization, fault injection, redundancy protection, or resource allocation;

[0026] 5. It is universally applicable to ResNet-like networks and other networks with main branch and shortcut branch fusion structures. Attached Figure Description

[0027] Figure 1 is a flowchart of a residual path-aware FPGA fault robust quantization perception training method.

[0028] Figure 2 is a flowchart of the residual addition node and the identification of the main branch path and shortcut branch path.

[0029] Figure 3 is a flowchart of path-level software fault sensitivity assessment.

[0030] Figure 4 is a flowchart of hardware exposure extraction, path-level risk modeling, and training scheduling. Detailed Implementation

[0031] The system in this embodiment includes a model parsing module, a residual path identification module, a software sensitivity assessment module, an HLS hardware exposure extraction module, a risk modeling module, and a training scheduling module.

[0032] The model parsing module reads the computation graph of the quantized neural network model. The residual path identification module identifies residual summing nodes and categorizes the input paths before these nodes into main branch paths and shortcut branch paths. The software sensitivity assessment module injects faults into specified paths and bits within the software environment and calculates the error rate. The HLS hardware exposure extraction module reads FIFO metadata, flow width, and depth information from the FPGA high-level synthesized dataflow representation. The risk modeling module calculates path-level risk indicators. The training scheduling module adjusts quantization-aware training based on these risk indicators.

[0033] In step S1, the quantized neural network model can be a model described by QONNX, ONNX, or other quantization operators. The model includes multiple residual blocks, each containing a main branch path and a shortcut branch path, which are fused at the residual summation node.

[0034] In steps S2 and S3, the system identifies the addition nodes in the computation graph. For the original quantization computation graph, the main branch path and shortcut branch path can be determined based on the production node, layer name, residual block number, or graph topology of the input tensor of the addition node. For the model that has been converted into an FPGA high-level synthesis data flow representation, the data flow addition core node can be identified, and the input flow of the node can be grouped according to its source.

[0035] In steps S4 and S5, bit-flipping fault injection is performed on the residual path in the software environment. For each residual layer... ,path and bits The quantized activation values ​​or intermediate tensors of the corresponding path are flipped on a preset sample set, and the number of prediction errors and the number of prediction class changes caused by faults are counted. Path-level software fault sensitivity can be calculated using the average conditional error rate of multiple bits and multiple random seeds.

[0036] In steps S6 to S8, the quantization model is converted into an FPGA / HLS data stream representation. The system extracts FIFO metadata from the data stream representation or HLS intermediate file, including FIFO depth, HLS data type, array parallelism, and stream name. For an array-type data word consisting of two 8-bit unsigned integers, the bit width of a single FIFO byte is 16 bits.

[0037] In step S9, path-level risk is calculated based on software fault sensitivity and hardware exposure. In step S10, the QAT training strategy is adjusted based on risk indicators, such as giving high-risk paths a higher fault injection probability, a larger robust loss weight, or a higher quantization protection strength.

[0038] Table 1 Symbol Explanation Table

[0039]

[0040] The software fault sensitivity of a path can be calculated using the following formula:

[0041]

[0042] in, For residual layer ,path Bits The number of samples that resulted in prediction errors after fault injection. The total number of samples injected for the corresponding fault.

[0043] Space hardware exposure can be calculated using the following formula:

[0044]

[0045] The time-based hardware exposure can be proxied using the maximum value of the FIFO depth corresponding to the path:

[0046]

[0047] The path-level joint risk index can be calculated using the following formula:

[0048]

[0049] In other embodiments, it is possible to , or Normalization can be performed, or path weights, bit weights, and hardware protection cost coefficients can be introduced.

[0050] Based on path-level risk indicators The training scheduling module can execute at least one of the following strategies:

[0051] 1. Increase the probability of bit-flipping fault injection for high-risk paths;

[0052] 2. Increase the weight of the robust loss term for higher-risk paths;

[0053] 3. Prioritize fault injection for highly sensitive bits in high-risk paths;

[0054] 4. Use higher bit widths, stricter pruning thresholds, or redundancy protection for high-risk paths;

[0055] 5. Reduce the frequency of fault injection for lower-risk paths to save training costs.

[0056] An example training loss is:

[0057]

[0058] in, For mission losses, For residual layer path The corresponding robustness constraint term, This is a normalized path-level risk indicator. This is the robustness loss coefficient.

[0059] In one embodiment, an 8-bit activation, 8-bit weight, and 16-bit bias quantized residual network with residual connections was selected as the model to be deployed. The model was trained using QAT and then exported as a QONNX / ONNX representation. Validation results showed that the maximum absolute difference between the PyTorch model and the QONNX model was 0, and the maximum absolute difference for the first 64 validation samples was also 0, indicating that the quantized model export link can be used for subsequent analysis.

[0060] Subsequently, software bit-flipping fault injection is performed on the main branch paths and shortcut branch paths of levels 1, 2, and 3 of the residual layer. The conditional error rate of each path is statistically analyzed under multiple bit positions and multiple random seeds, and the average is taken to obtain the software fault sensitivity.

[0061] Next, the quantization model is input into the FPGA / HLS data stream generation process to obtain a data stream representation containing HLS node and FIFO metadata. In this representation, the three residual adder cores are StreamingAdd_0_hls, StreamingAdd_1_hls, and StreamingAdd_2_hls. The system reads the input stream of each residual adder core and groups it into main paths and shortcut paths according to the input source. For each path, the system extracts the FIFO depth, HLS type, array parallelism, and byte bit width from the FIFO metadata, and calculates... and .

[0062] The experiment yielded the following results:

[0063] Table 2 Summary of Path-Level Risks

[0064]

[0065] As the results above show, all three highest-risk paths are shortcut paths. Although shortcut paths may appear simpler in the software computation graph, in the FPGA dataflow implementation, their alignment with the main branch path may introduce a larger FIFO depth and a higher residence time proxy, thus exhibiting a higher joint risk. Therefore, this embodiment supports differentiated allocation of fault injection strength and robust loss weights during training according to path-level risk indicators.

Claims

1. A residual path-aware FPGA fault robust quantization perception training method, characterized in that, Includes the following steps: (1) Obtain the quantized neural network model to be deployed to the FPGA. The quantized neural network model includes at least one residual connection structure. Each residual connection structure includes a main branch path and a shortcut branch path, and they are fused at the residual addition node. (2) Identify residual addition nodes in the computation graph of the quantized neural network model, and determine the main branch path and shortcut branch path before each residual addition node respectively; (3) Perform software fault injection on the main branch path and the shortcut branch path respectively to obtain path-level software fault sensitivity. ,in Indicates a residual layer or residual block. Indicates the path type; (4) Convert the quantized neural network model into an FPGA high-level synthesis dataflow representation, and extract the hardware exposure information corresponding to the input path of each residual addition node from the high-level synthesis dataflow representation. The hardware exposure information includes at least the spatial exposure amount. and time exposure ; (5) Based on the software fault sensitivity Spatial exposure and time exposure Construct path-level risk indicators ; (6) Based on the path-level risk indicators Adjusting the fault injection probability, robust loss weight, quantization protection strength, or combinations thereof during the quantization perception training process allows high-risk residual paths to obtain higher fault robustness constraints during training.

2. The method according to claim 1, characterized in that, In step S2, the residual addition node is an addition node in the computation graph of the general intermediate layer protocol ONNX or QONNX, or a data stream addition core node obtained after high-level synthesis and conversion by the FPGA. If residual paths are identified in the high-level synthesized data flow representation of the FPGA, multiple input streams connected to the same data flow adder core node are grouped by source, and the main branch path and shortcut branch path are determined respectively.

3. The method according to claim 1, characterized in that, In step S3, software fault injection includes: performing bit flipping on the quantization activation value or intermediate tensor of a specified residual layer, a specified path and a specified bit on a preset sample set, and statistically analyzing the classification error rate, conditional error rate, predicted class change rate or a combination thereof caused by the bit flipping.

4. The method according to claim 1, characterized in that, The software fault sensitivity The results are obtained based on fault injection results from multiple bits and multiple random seeds, and include at least one of the following: average path conditional error rate, highest sensitive bit error rate, and bit-weighted path error rate.

5. The method according to claim 1, characterized in that, In step S4, the spatial exposure amount The data is calculated based on the depth of the FIFO or data stream buffer corresponding to the path, the number of parallel arrays, and the bit width of a single data word. When the FIFO data type is an array-type high-level synthesis type, the bit width of a single data word is determined by the product of the bit width of the array element and the number of array elements.

6. The method according to claim 1, characterized in that, In step S4, the time exposure amount The path-corresponding FIFO depth, the path maximum FIFO depth, the high-level synthesis tool (HLS) scheduling cycle, the register transfer level (RTL) simulation dwell cycle, or a combination thereof, are used as time exposure proxies.

7. The method according to claim 1, characterized in that, In step S5, the path-level risk index is calculated according to the following formula. Alternatively, a normalization coefficient, path weight coefficient, bit weight coefficient, or hardware protection cost coefficient can be introduced based on the above formula.

8. The method according to claim 1, characterized in that, In step S6, at least one of the following training adjustments is performed on the high-risk paths: increasing the bit flip injection probability of the path; increasing the weight of the corresponding loss term of the path; prioritizing fault injection of the highly sensitive bits of the path; and adopting higher precision quantization, stricter pruning thresholds, or redundancy protection strategies for the path.

9. The method according to claim 1, characterized in that, The quantized neural network model is a convolutional neural network with residual connections, a residual network, a lightweight residual network, a residual substructure in an object detection network, or other neural networks with a fusion structure of main branches and shortcut branches.

10. A residual path-aware FPGA fault robust quantization perception training system, characterized in that, include: The system comprises a model parsing module, a path identification module, a software sensitivity assessment module, a hardware exposure extraction module, a risk modeling module, and a training scheduling module; wherein each module is used to execute the method described in any one of claims 1 to 9.

Images