A full-stack NPU system supporting multi-level fault mitigation mechanisms

Abstract

The present application discloses a full-stack NPU system that supports a multi-level fault mitigation mechanism, including: multiple NPU functional blocks for performing neural network computing tasks; multiple hardware protection modules for providing error detection and correction capabilities for at least one NPU functional block within the NPU or the data path within the NPU; and a control mechanism for coordinating the operations of multiple hardware protection modules. Utilizing the solution of the present application, NPU functional block computing errors or data path transmission errors caused by hardware failures, etc. can be detected and corrected in a timely manner, preventing erroneous calculation results from being continued to be used or output, and ensuring the accuracy of the final results. At the same time, even if some hardware fails, the error correction mechanism can cover up the error to a large extent, maintain the normal operation of the system, improve the overall availability of the system and the task success rate, thereby significantly improving the reliability and robustness of the NPU.

Description

Technical Field

[0001] The present application generally relates to the field of hardware architecture design and reliability assurance technology for neural network processors. More specifically, the present application relates to a full-stack NPU system that supports a multi-level fault mitigation mechanism. Background Art

[0002] With the rapid development of deep learning technology in recent years, deep neural networks (DNNs) have become a core technology driving progress in many key areas, including computer vision, natural language processing, and autonomous driving. DNN models, especially common large language models, require enormous computational complexity during inference (for example, a single inference run can involve hundreds of millions of multiplication-addition operations per frame) and a massive number of parameters (millions to hundreds of millions), placing extremely high demands on computing platform performance.

[0003] To meet the computing power and data throughput requirements of DNN inference, modern system-on-chip (SoC) designs commonly integrate dedicated neural network processors (NPUs). NPUs typically include dedicated hardware resources such as on-chip memory (such as scratchpads), tensor processing units (such as systolic arrays), and activation function hardware modules. These resources are designed to efficiently execute forward computations of DNN models, significantly improving inference efficiency.

[0004] However, the highly parallel and computationally intensive nature of NPUs, while improving performance, also makes them extremely sensitive to hardware failures, significantly amplifying their impact on system reliability. Related research indicates that NPUs have become a major challenge in modern SoC reliability design. Existing technologies face the following challenges and limitations in addressing NPU reliability: First, the architecture of modern DNN models is becoming increasingly complex; second, transistor size continues to shrink, the number of integrated circuits continues to increase, and operating voltages are gradually decreasing; third, aggressive hardware optimization techniques have resulted in extreme compression of hardware operating margins; and fourth, hardware architectures rely heavily on data reuse mechanisms, which often propagate single failures to multiple output elements.

[0005] In view of this, there is an urgent need to provide a full-stack NPU system that supports a multi-level fault mitigation mechanism to provide efficient, flexible, adaptable and comprehensive reliability assurance for the NPU. Summary of the Invention

[0006] In order to at least solve one or more technical problems mentioned above, the present application proposes a full-stack NPU system that supports a multi-level fault mitigation mechanism in multiple aspects.

[0007] The present application provides a full-stack NPU system that supports a multi-level fault mitigation mechanism, the system comprising: multiple NPU functional blocks for performing neural network computing tasks; multiple hardware protection modules for providing error detection and correction capabilities for at least one NPU functional block within the NPU or a data path within the NPU; and a control mechanism for coordinating the operations of the multiple hardware protection modules; wherein the control mechanism uses an instruction set architecture containing dedicated instructions to trigger or manage the fault-tolerant operations of the multiple hardware protection modules.

[0008] In some embodiments, the plurality of NPU functional blocks include at least: a control unit and an instruction reservation station for processing instructions; a local memory for storing data; a systolic array for performing core matrix operations; and a nonlinear operator unit for performing nonlinear operations.

[0009] In some embodiments, the multiple hardware protection modules include a first protection module, which is used to protect data stored in the register of the corresponding NPU functional block through an ECC mechanism; wherein the ECC mechanism is a fully monitored ECC constructed based on a multi-level parity tree; the ECC mechanism includes multiple partial odd check bits and a total odd check bit; the multiple partial odd check bits are generated by grouping the original data bits stored in the register, and the grouping is based on the binary index of the original data bit itself; each partial odd check bit is used to monitor the set of original data bits whose corresponding bits in the binary index of the original data bit itself are 1; the total odd check bit is generated by aggregating the results of the multiple partial odd check bits; when an error occurs in the data stored in the register of the corresponding NPU functional block, single error positioning and multi-bit error judgment are performed through the deviations corresponding to each partial odd check bit and the deviation corresponding to the total odd check bit.

[0010] In some embodiments, in the process of protecting data stored in the register of the corresponding NPU functional block through the ECC mechanism, the following steps are performed: when an error occurs in the data stored in the register of the corresponding NPU functional block, it is determined whether the partial odd parity bits and the total odd parity bits are deviated; when there are deviations in some odd parity bits and the total odd parity bits are deviated, single-bit error positioning is performed; when there are deviations in some odd parity bits but the total odd parity bits are not deviated, it is determined that multiple-bit errors have occurred; and error correction is performed based on the located error bits and the odd parity principle.

[0011] In some embodiments, the multiple hardware protection modules also include a second protection module, which is used to protect the integrity of the data in the local memory; the second protection module includes: a checksum adder, which is used to calculate one or more original checksums based on the data to be written when the data is written to the local memory; a protection register, which is located in the local memory and is used to store the original checksum; a protection linker, including a link block and an error block, which is used to work synchronously with the checksum adder when the data is written, allocate a storage area for the data to be written, and establish an association between the position of the data to be written in the local memory and the storage position of its corresponding original checksum in the protection register; the error block is used to record the a history of errors in a local memory and a specific location of the systolic array; a data validator for, when reading data from the local memory, recalculating one or more current checksums based on the read data, obtaining an original checksum associated with the read data and stored in the protection register through the protection linker, and comparing the recalculated current checksum with the obtained original checksum to detect whether there is a mismatch, wherein the mismatch refers to a numerical difference between the current checksum and the original checksum; and a data error corrector for being activated when the data validator detects the mismatch, and performing differentiated error handling based on checksum difference information indicated by the detected mismatch.

[0012] In some embodiments, during the differentiated error handling process performed by the data corrector, the following steps are performed: determining whether the mismatch is a mismatch between the row checksum of a certain row and the column checksum of the corresponding column; in response to the mismatch not being a mismatch between the row checksum of a certain row and the column checksum of the corresponding column, suppressing activation of the data corrector and not performing error correction operations; in response to the mismatch being a mismatch between the row checksum of a certain row and the column checksum of the corresponding column, determining whether there is a consistency deviation; in response to the existence of a consistency deviation, performing cross-positioning to locate a single erroneous data bit, the data corrector corrects the single erroneous data bit through the consistency deviation value, and updates the corresponding error record in the error block; in response to the absence of a consistency deviation, no action is performed.

[0013] In some embodiments, when the checksum adder generates a checksum for data written to the local memory, the checksum adder selectively calculates the original checksum only for key bits in the data word of the data to be written according to the data type and preset sensitivity.

[0014] In some embodiments, the multiple hardware protection modules further include a third protection module, the third protection module being configured to protect the computational integrity of the systolic array during the execution of a core matrix multiplication and addition operation through an algorithmic fault-tolerant logic; the third protection module including a shield array, the shield array employing a plurality of adder trees and a set of multipliers aligned with the number of processing units in each row of the systolic array; wherein, during the operation of the systolic array, at least the following steps are performed: before the core matrix multiplication and addition operation of the systolic array begins or during the data loading phase, the data verifier and the data error corrector are used to perform data verification and error correction on the input matrices involved in the operation, and the checksum adder is used to calculate an initial checksum vector corresponding to the input matrix according to the algorithmic fault-tolerant logic; when the systolic array performs its core matrix multiplication and addition operation to calculate a result matrix, the shield array calculates an expected checksum vector corresponding to the result matrix according to the algorithmic fault-tolerant logic using the initial checksum vector and the data input to the systolic array; when the systolic array generates a calculation result, the data verifier is used to Comparing the expected checksum vector with the checksum vector calculated based on the actual output of the systolic array to determine whether there is a mismatch; in response to no mismatch, not performing any action; in response to the presence of a mismatch, determining whether the mismatch satisfies a preset error correction condition; in response to the mismatch satisfying the preset error correction condition, triggering the data error corrector to perform corresponding error correction; in response to the mismatch not satisfying the preset error correction condition, not performing any action; wherein, in the process of comparing the expected checksum vector with the checksum vector calculated based on the actual output of the systolic array, when a continuous row or column mismatch is detected, analyzing the matrix position corresponding to the checksum vector where the mismatch first occurs in the continuous mismatch sequence, inferring a fault processing unit within the systolic array that causes the continuous row or column mismatch based on the matrix position, and marking the fault processing unit in the error block; and updating the area of the local memory where the mismatched data is located or the area of the tile of the systolic array where the mismatched data is located based on the mismatch detected by the data validator.

[0015] In some embodiments, a multi-stage pipeline architecture is employed during operation of the systolic array, and overlapping execution of pipeline stages is achieved through the multi-stage pipeline architecture. During the overlapping execution of pipeline stages through the multi-stage pipeline architecture, the following steps are performed: a stage before a core matrix multiplication-addition operation of the systolic array begins is overlapped in time with a stage in which the systolic array performs its core matrix multiplication-addition operation to calculate a result matrix; and a stage after the systolic array generates a calculation result is overlapped in time with a data loading stage of the systolic array.

[0016] In some embodiments, the multiple hardware protection modules further include a fourth protection module, which is used to verify the operation result of the nonlinear operator unit based on the mathematical characteristics of the nonlinear operator executed by the nonlinear operator unit or through a hardware redundancy and output comparison mechanism; wherein, when the nonlinear operator executed by the nonlinear operator unit belongs to the first type of nonlinear operator, the fourth protection module verifies the operation result of the nonlinear operator unit based on the mathematical characteristics of the nonlinear operator executed by the nonlinear operator unit. In the process of the fourth protection module verifying the operation result of the nonlinear operator unit based on the mathematical characteristics of the nonlinear operator executed by the nonlinear operator unit, the fourth protection module performs at least one of the following steps: The fourth protection module comprises the following steps: checking whether the sum of all output values after LayerNorm normalization and offset is zero, whether the ReLU output is non-negative, whether the positive output position corresponds to the input position, and whether the Softmax output sum is one; when the nonlinear operator executed by the nonlinear operator unit belongs to the second type of nonlinear operator, the fourth protection module verifies the operation result of the nonlinear operator unit through the hardware redundancy and output comparison mechanism. In the process of the fourth protection module verifying the operation result of the nonlinear operator unit through the hardware redundancy and output comparison mechanism, the fourth protection module retains multiple calculation copies of the corresponding nonlinear operator, and compares the operation results generated by each of them one by one to detect errors in the operation process.

[0017] Through the full-stack NPU system supporting a multi-level fault mitigation mechanism as provided above, the embodiments of the present application provide error detection and correction capabilities for at least one NPU functional block or data path within the NPU through multiple hardware protection modules. By coordinating the operation of multiple hardware protection modules through a control mechanism, it is possible to promptly detect and correct NPU functional block calculation errors or data path transmission errors caused by hardware failures, etc., preventing erroneous calculation results from being used or output, and ensuring the accuracy of the final result. Even if partial hardware failure occurs, the error correction mechanism can largely mask the error and maintain normal system operation, improving the overall system availability and task success rate, thereby significantly enhancing the reliability and robustness of the NPU. Furthermore, the use of multiple protection modules allows for the design of specialized, most effective protection mechanisms for different functional units or different types of errors within the NPU. Furthermore, protection modules can be flexibly deployed based on the critical paths and error-prone links of the NPU architecture, achieving focused protection of key NPU functional blocks and data paths while achieving broader coverage of the entire unit. Furthermore, the control mechanism intelligently schedules and manages the activation, operation, and information exchange of each protection module, avoiding redundant detection and optimizing the protection process.

[0018] Furthermore, in some embodiments, an ECC mechanism is employed in the first protection module to protect data stored in registers of the corresponding NPU functional blocks, using multiple odd parity bits to cover all relevant bits in the registers, thereby ensuring high register reliability. By configuring the ECC mechanism as a fully monitored ECC based on a multi-level parity tree, precise bit-level error detection and correction can be achieved using minimal redundant bits, significantly enhancing the reliability of both the control and storage paths.

[0019] Furthermore, in some embodiments, in the process of protecting the integrity of data in the local memory by a second protection module including a checksum adder, a protection register, a protection linker, a data verifier, and a data error corrector, the data verifier, in combination with the checksum adder and the protection register, can quickly detect whether there is an error when reading the data. After detecting an error, the data error corrector can use the checksum information to automatically repair errors within a certain degree to prevent erroneous data from being used. This reduces the risk of computing failure or system crash due to data corruption by automatically processing memory data errors at the hardware level, significantly improving the stable operation capability of the NPU in complex environments. Thus, a hardware-level, dedicated data integrity protection mechanism is provided for the local memory inside the NPU to ensure that the data stored therein does not undergo unexpected changes during reading, writing, and retention.

[0020] Furthermore, in some embodiments, a third protection module employs algorithmic fault-tolerant logic to protect the computational integrity of the systolic array during core matrix multiplication and addition operations. The shield array computes the expected checksum in parallel with the systolic array, rather than waiting for the final result before checking. This significantly reduces the performance latency associated with protection. The shield array utilizes hardware resources compatible with the systolic array, allowing for partial design reuse or leveraging similar processing units for checksum calculations. This reduces hardware overhead compared to fully redundantly replicating the entire systolic array. Input validation, error correction, and initial checksum calculations are performed during the data loading phase, utilizing pipelines to hide the latency of some protection operations. Error correction is initiated only when a mismatch meeting specific conditions is detected, avoiding unnecessary error correction overhead while enabling the correction of some computational errors and improving fault tolerance.

[0021] Furthermore, in some embodiments, the fourth protection module verifies the operation results of the nonlinear operator unit based on the mathematical characteristics of the nonlinear operator executed by the nonlinear operator unit or through hardware redundancy and output comparison mechanism. By combining the two strategies, the module can provide effective protection for different types of nonlinear operators, ensuring the overall reliability of the nonlinear processing link, and providing a set of differentiated, operator characteristic-based result verification mechanisms for the nonlinear operator units in the NPU, ensuring the correctness of these nonlinear transformation operation results that have an important impact on model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other objects, features and advantages of the exemplary embodiments of the present application will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0023] Figure 1 An exemplary structural block diagram of a full-stack NPU system supporting a multi-level fault mitigation mechanism according to an embodiment of the present application is shown;

[0024] Figure 2 An exemplary structural block diagram of the systematic framework Strix according to an embodiment of the present application is shown;

[0025] Figure 3 An exemplary structural block diagram showing the specific composition of multiple hardware protection modules according to an embodiment of the present application is shown;

[0026] Figure 4 An exemplary structural block diagram showing the working process of the second protection module in an embodiment of the present application is shown;

[0027] Figure 5 An exemplary flow chart showing the differential error processing performed by the data error corrector according to an embodiment of the present application is shown;

[0028] Figure 6 A schematic diagram comparing obtaining an A×B checksum through a shield array and obtaining an A×B checksum through a traditional ABFT in an embodiment of the present application is shown;

[0029] Figure 7 An exemplary flow chart showing the specific working process of the shield array after the pulsating array generates calculation results according to an embodiment of the present application;

[0030] Figure 8 A schematic diagram of a multi-stage pipeline architecture according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0032] It should be understood that the terms "include" and "comprising" used in the description and claims of this application indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0033] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this specification and claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" as used in this specification and claims refers to any and all possible combinations of one or more of the associated listed items, including and including these combinations.

[0034] In existing technologies, addressing NPU reliability issues mainly faces the following challenges:

[0035] First, there are the limitations of the inherent fault tolerance of neural networks: Although artificial neural networks theoretically possess a certain degree of fault tolerance, and early studies have shown that model outputs may not change much at low bit error rates, this "statistical robustness" is difficult to guarantee stability in the actual NPU hardware execution environment, especially when processing critical tasks. When NPU hardware executes complex models, its sensitivity to different types of faults (such as control path errors, memory bit flips, and computing unit failures) increases significantly. Errors can propagate and accumulate within the NPU (such as data reuse in storage paths and computing arrays). When the error rate exceeds a certain threshold or occurs in critical data bits (such as the sign bit or high-order exponent bit of a floating-point number), it can cause a sharp drop in model accuracy or even cause inference failure.

[0036] Second, software-level fault-tolerance methods are inadequate: Existing software-level fault-tolerance methods, such as instruction redundancy (IR), fault-injection training, and static hardening strategies based on sensitive layers, have significant shortcomings. While instruction redundancy is flexible, it often requires extensive manual optimization, significantly increases computational latency (possibly doubling it), and can typically only detect errors but not locate or correct them. Fault-injection training primarily improves the model's tolerance to simulated errors during training but cannot effectively detect and handle actual hardware faults during runtime, making it difficult to meet the requirements of functional safety standards such as ISO-26262. Static hardening based on sensitive layers lacks versatility and struggles to adapt to the dynamic changes of online tasks.

[0037] Third, the application limitations of hardware-level fault tolerance technology: Hardware-level fault tolerance mechanisms are currently the mainstream, but existing technologies such as parity check / error detection code (ECC) and triple modular redundancy (TMR) each have limitations. ECC is primarily used in storage and registers, with limited multi-bit error detection capabilities, making it difficult to correct or accurately locate errors. Furthermore, tight integration with complex NPU data paths (such as systolic arrays) is challenging. While TMR offers high fault tolerance, its significant area and power consumption overhead (which can exceed 200%) makes it unsuitable for resource-constrained edge computing or embedded platforms. Even solutions that partially utilize TMR (such as protecting only the compute array) cannot cover the entire data path, leaving reliability "holes."

[0038] Fourth, there is a lack of full-stack hardware-software collaborative fault-tolerance design: Most current reliability strategies focus solely on a single hardware layer (e.g., storage, compute) or specific failure modes (e.g., transient errors), lacking systematic, cross-hardware collaborative design. Existing technology systems lack support for hardware-software collaborative instruction sets (e.g., instruction extensions for error reporting and data repair), efficient binding mechanisms for data and its verification information, dynamic scheduling capabilities for fault-tolerance strategies in real-time systems (e.g., adjusting the fault-tolerance level based on load to balance performance and reliability), and error traceability and fault-tolerance optimization mechanisms in high-reliability scenarios.

[0039] In view of this, the embodiment of the present application provides a full-stack NPU system that supports a multi-level fault mitigation mechanism. By introducing a checksum adder and a protection linker, dynamic binding and continuous monitoring of the original data are achieved, and the system has the ability to quickly detect, locate and record the source of hardware errors, significantly alleviating the problem of error propagation being difficult to track. By designing a dedicated instruction set to directly drive multiple hardware protection modules, it supports automatic triggering of detection and error correction operations, simplifies fault-tolerant deployment logic, and significantly reduces software development and system maintenance costs. By constructing a fully monitored ECC mechanism based on a multi-level parity tree, accurate detection and correction of bit-level errors can be achieved using very few redundant bits, significantly enhancing the reliability of the control path and storage path.

[0040] At the same time, by sinking the ABFT principle into the hardware layer and combining it with a shield array and pipeline scheduling mechanism, the ABFT algorithm is executed in situ at the hardware level, ensuring error detection and dynamic verification without computational delay. By constructing corresponding verification mechanisms based on the mathematical characteristics of different nonlinear operators and performing bit-level verification of ReLU negative value filtering results, extremely low-cost but effective guarantee of the correctness of computational results is achieved. By introducing multi-stage pipeline overlap, asynchronous error correction, and dynamic selective deployment mechanisms, combined with a hierarchical slow factor adjustment algorithm in task scheduling, the fault tolerance mechanism is dynamically adjusted according to different error sensitivities, effectively controlling performance loss while balancing reliability and schedulability.

[0041] Figure 1 An exemplary structural block diagram of a full-stack NPU system 100 supporting a multi-level fault mitigation mechanism according to an embodiment of the present application is shown.

[0042] like Figure 1 As shown, the full-stack NPU system 100 includes multiple NPU functional blocks 110, multiple hardware protection modules 120, and a control mechanism 130. Specifically, the multiple NPU functional blocks 110 are used to perform neural network computing tasks. The multiple hardware protection modules 120 are used to provide error detection and correction capabilities for at least one NPU functional block in the multiple NPU functional blocks 110 or the data path within the NPU. The control mechanism 130 is used to coordinate the operations of the multiple hardware protection modules 120.

[0043] In an embodiment of the present application, a systematic framework Strix is provided to integrate the aforementioned full-stack NPU system 100, so that the full-stack NPU system 100 has the ability to detect, locate and correct hardware faults.

[0044] Figure 2 An exemplary structural block diagram of the systematic framework Strix according to an embodiment of the present application is shown.

[0045] like Figure 2 As shown, multiple NPU functional blocks 110 interact with an external CPU and main memory system via interfaces and direct memory access (DMA) controllers. Multiple hardware protection modules 120 provide corresponding protection for the multiple NPU functional blocks 110. The CPU receives tasks sent by the operating system. Multiple NPU functional blocks 110 include a control unit 111, an instruction reservation station 112, a local memory 113, a systolic array 114, and a nonlinear operator unit 115.

[0046] Specifically, the control unit 111 and the instruction reservation station 112 are used to process instructions. The instruction reservation station 112 receives instructions (such as storage instructions, calculation instructions, load instructions, configuration instructions, etc.) from the interface and transmits the instructions to the control unit 111. The control unit 111 is responsible for decoding instructions and generating control signals by scheduling the storage controller, load controller, calculation controller, and computing module controller.

[0047] Specifically, local memory 113 is used to store data. It includes a temporary register, an accumulator, a protection register, and a protection linker. The temporary register is used to store input data and weight data loaded from main memory. The accumulator is used to store intermediate or final results generated by systolic array calculations. The protection register is used to store checksums (including row checksums and column checksums) corresponding to the data in the temporary register and accumulator. The protection linker is located between the direct memory access controller and local memory 113 and is used to associate the original data addresses in the temporary register and accumulator with their checksum addresses in the protection register.

[0048] Specifically, the systolic array 114 is used to perform core matrix operations. It is the core computing engine, composed of a large number of computing modules, and efficiently performs matrix multiplication and addition operations. The systolic array 114 reads inputs and weights from the local memory 113 and writes the results back to the accumulator.

[0049] Specifically, the nonlinear operator unit 115 is used to perform nonlinear operations. More specifically, the nonlinear operator unit 115 is responsible for performing transposition operations through a transposer, and performing nonlinear operations such as activation functions (ReLU), normalization (LayerNorm), and Softmax.

[0050] In an embodiment of the present application, the multiple NPU functional blocks 110 may include other units in addition to the control unit 111, the instruction reservation station 112, the local memory 113, the systolic array 114 and the nonlinear operator unit 115, and the present application does not limit them here.

[0051] In the embodiment of the present application, the specific composition of the multiple hardware protection modules 120 can be found in Figure 3 .

[0052] Figure 3 An exemplary structural block diagram showing the specific composition of multiple hardware protection modules in an embodiment of the present application is shown.

[0053] like Figure 3 As shown, the multiple hardware protection modules 120 include a first protection module 121 , a second protection module 122 , a third protection module 123 and a fourth protection module 124 .

[0054] In an embodiment of the present application, the first protection module 121 is configured to protect data stored in registers of corresponding NPU functional blocks through an ECC mechanism.

[0055] In an embodiment of the present application, the ECC mechanism is a fully monitored ECC constructed based on a multi-level parity tree.

[0056] In the NPU, registers are used to store and transmit instructions, constant data, and other key control information. Their reliability is directly related to the stability of the system. Once an error occurs in instruction decoding, it may lead to operation errors, task failures, or even program crashes. Unlike traditional CPUs, NPUs often need to process large-scale, packaged compound instructions. For example, the scratchpad write instruction in Gemmini is as large as 382 bits. Such a large data scale poses a huge challenge to traditional ECC methods, which often have limitations in error detection and correction capabilities. Therefore, a fully monitored ECC is used for registers used to store instructions, control signals, configuration information, etc. in the control unit 111 and the instruction reservation station 112, as well as registers for storing nonlinear operator constants in the nonlinear operator unit 115, etc., using multiple odd parity bits to cover all relevant bits in the corresponding registers, thereby ensuring the high reliability of the registers.

[0057] In an embodiment of the present application, the Hamming distance of the ECC mechanism is 2, and [log2(α+1)]+1 bits are introduced (where α represents the register width) to improve the reliability of key registers.

[0058] In an embodiment of the present application, the ECC mechanism includes multiple partial odd parity bits and a total odd parity bit. The multiple partial odd parity bits are generated by grouping the original data bits stored in the register, and the grouping is based on the binary index of the original data bit itself. Each partial odd parity bit is used to monitor the set of original data bits whose corresponding bits in the binary index of the original data bit itself are 1. The total odd parity bit is generated by aggregating the results of the multiple partial odd parity bits. When an error occurs in the data stored in the register of the corresponding NPU functional block, single error location and multi-bit error judgment are performed through the deviations corresponding to the respective partial odd parity bits and the deviation corresponding to the total odd parity bit.

[0059] When an error occurs in the data stored in the registers of the corresponding NPU functional block while using the ECC mechanism to protect the data stored in the registers of the corresponding NPU functional block, a determination is made as to whether all partial odd parity bits and the total odd parity bits deviate. If both partial odd parity bits and the total odd parity bits deviate, a single-bit error is located. If both partial odd parity bits and the total odd parity bits do not deviate, a multi-bit error is determined. Error correction is then performed based on the located error bits and the odd parity principle.

[0060] Specifically, when a partial odd parity bit deviates, and the total odd parity bit deviates, a single-bit error can be inferred. This is because if a data bit flips (from 0 to 1 or 1 to 0), the calculated results of all partial odd parity bits containing that data bit will change, causing a deviation from the stored (expected) parity bit. Therefore, a single data bit error will cause a deviation in a specific set of partial odd parity bits. The total odd parity bit is generated based on the results of all partial odd parity bits and deviates only if an odd number of errors occurs. Therefore, a single data bit error (an odd number of errors) will inevitably cause a deviation in the corresponding partial odd parity bit and the total odd parity bit.

[0061] Specifically, when some odd parity bits are deviated but the total odd parity bits are not deviated, it can be inferred that multiple bit errors have occurred. This is because multiple bits may cause deviations in some odd parity bits, but because multiple errors may cancel each other out, the total odd parity bits may still be correct.

[0062] In one embodiment of the present application, the data in the register is 15-bit data 100110000110011. Since 15 requires a 4-bit binary number to represent, a 4-bit binary index unique to the position of each data bit of the data is represented by a 4-bit binary number (for example, the index of data bit 1 is 0001, and the index of data bit 15 is 1111). The 15 data bits are represented according to their specific positions in the binary index (from right to left, or in 2 0 , 2 1 , 2 2 , 2 3 The first partial odd parity bit monitors the first bit of all binary indices (the rightmost bit, 2 0 bit) is 1, the second partial odd parity bit monitors all binary index 2 bits (2 1 bit) is 1, the third partial odd parity bit monitors all binary index 3 bits (2 2 bit) is 1, the fourth partial odd parity bit monitors all binary index 4 bits (leftmost, 2 3 bit) is a data bit that is 1.

[0063] When the values of the first, second, third, and fourth partial odd parity bits are all 0, this means that the number of data bits that are 1 monitored by each of them is an even number (due to odd parity, a result of 0 indicates an even number of 1s). When the number of data bits that are 1 monitored by a partial odd parity bit is odd, the value of that partial odd parity bit is 1. When the value of the total odd parity bit is 0, it means that the total number of data bits that are 1 monitored by the first, second, third, and fourth partial odd parity bits is even. When the value of the total odd parity bit is 1, it means that the total number of data bits that are 1 monitored by the first, second, third, and fourth partial odd parity bits is odd.

[0064] When an error occurs in a register data bit, the values of all partial odd parity bits monitoring that bit change (from 0 to 1 or from 1 to 0, deviating from the expected value). Combining the index positions corresponding to these changed partial odd parity bits and the change in the total odd parity bit yields the binary index of the erroneous data bit. For example, if the values of both the first and second partial odd parity bits deviate from the expected value, and the total odd parity bit also deviates from the expected value, this indicates that the error occurred in a data bit where both bits 1 and 2 of the binary index are 1. This combination is 0011, corresponding to data bit 3.

[0065] By using the ECC mechanism to protect the data stored in the registers of the corresponding NPU functional blocks, compared with the traditional parity-check ECC, it can not only achieve error detection, but also support error location and correction, and has extremely low redundancy overhead.

[0066] In the embodiment of the present application, the second protection module 122 is used to protect the integrity of data in the local memory 113 .

[0067] In the embodiment of the present application, the specific composition of the second protection module 122 can be found in Figure 4 .

[0068] Figure 4 An exemplary structural block diagram of the working process of the second protection module in an embodiment of the present application is shown.

[0069] like Figure 4 As shown, the second protection module 122 includes a checksum adder 1221 , a protection register 1222 , a protection linker 1223 , a data verifier 1224 and a data error corrector 1225 .

[0070] In an embodiment of the present application, the checksum adder 1221 is used to calculate one or more original checksums based on the data to be written when writing data into the local memory 113. Specifically, the original checksums include row checksums and column checksums.

[0071] In an embodiment of the present application, the checksum adder 1221 is coupled with a set of adder trees to generate row checksums, while a set of adder-register units are used to continuously accumulate and generate column checksums.

[0072] In embodiments of the present application, when the checksum adder 1221 generates a checksum for data written to the local memory 113, the checksum adder 1221 selectively calculates the raw checksum for only the key bits in the data word of the data to be written, based on the data type and preset sensitivity. For example, in some embodiments, the checksum is generated only for the sign bit and exponent bit in the data word. This can reduce deployment overhead.

[0073] In the embodiment of the present application, the protection register 1222 is located in the local memory 113 and is used to store the aforementioned original checksum.

[0074] In an embodiment of the present application, the protection linker 1223 includes a link block and an error block.

[0075] Specifically, the link block is used to work synchronously with the checksum adder 1221 when writing data, allocate a storage area for the data to be written, and establish an association between the location of the data to be written in the local memory 113 and the storage location of its corresponding original checksum in the protection register.

[0076] Specifically, the error block is used to record the error history of specific locations in the local memory 113 and the systolic array 114. For example, it is configured to record the error that occurs in each data row in the local memory 113 and the ID of the tile in the systolic array 114 where the error occurs, and to count the number of detected errors.

[0077] In an embodiment of the present application, the data validator 1224 is used to recalculate one or more current checksums based on the read data when reading data from the local memory 113, obtain the original checksum associated with the read data and stored in the protection register through the protection linker 1223, and compare the recalculated current checksum with the obtained original checksum to detect whether there is a mismatch, wherein the aforementioned mismatch refers to a numerical difference between the current checksum and the original checksum.

[0078] In an embodiment of the present application, the current checksum includes a row checksum and a column checksum.

[0079] In an embodiment of the present application, the data error corrector 1225 is configured to be activated when the data error corrector 1225 detects the aforementioned mismatch, and perform differentiated error processing through the checksum difference information indicated by the detected mismatch.

[0080] In the embodiment of the present application, the specific process involved in the data error corrector 1225 performing differentiated error processing can be found in Figure 5 .

[0081] Figure 5 An exemplary flowchart of a data error corrector performing differentiated error processing according to an embodiment of the present application is shown.

[0082] like Figure 5 As shown, in step S510, it is determined whether the mismatch is that both the row checksum of a certain row and the column checksum of the corresponding column do not match. In response to the mismatch not being that both the row checksum of a certain row and the column checksum of the corresponding column do not match, in step S520, the activation of the data error corrector is suppressed and the error correction operation is not performed. In response to the mismatch being that both the row checksum of a certain row and the column checksum of the corresponding column do not match, in step S530, it is determined whether there is a consistency deviation. In response to the presence of a consistency deviation, in step S540, a single erroneous data bit is located by performing cross positioning, and the data error corrector corrects the single erroneous data bit by using the consistency deviation value, and updates the corresponding error record in the error block. In response to the absence of a consistency deviation, in step S550, no action is performed.

[0083] In an embodiment of the present application, when the row checksum of a row and the column checksum of the corresponding column do not match, the numerical deviation reflected by the two mismatches is further checked (i.e., the difference between the stored original checksum and the recalculated current checksum). If the checksum deviation of row i is the same as the checksum deviation of column j or conforms to the expected pattern caused by a single bit error, it is a consistency deviation. Next, cross-positioning is performed: by identifying the row index and column index where there is a mismatch, the source of the error is uniquely determined to be the data bit located at the intersection of row i and column j. Subsequently, the data error corrector 1225 uses the numerical value of the consistency deviation to infer the value of the erroneous bit and perform correction.

[0084] In an embodiment of the present application, when the data validator detects that only one row checksum mismatches while all column checksums match, or detects that only one column checksum mismatches while all row checksums match, it determines that the single mismatch is more likely to be caused by an error in the checksum value itself during storage or transmission, rather than an error in the original data. As a result, the activation of the data error corrector 1225 is suppressed, and error correction operations are not performed on potentially fault-free data, thereby avoiding false positives and unnecessary processing overhead.

[0085] The second protection module 122 can detect and correct memory errors, significantly improving the reliability and robustness of NPU operations. The checksum adder 1221 calculates and stores a checksum when data is written to local memory. The data verifier 1224 recalculates and compares the checksum with the original checksum when data is read, thereby detecting data errors that may occur during storage. When the data verifier 1224 detects a mismatch, the data error corrector 1225 locates the specific error bit based on a specific checksum mismatch pattern (simultaneous row and column checksum mismatch and consistency deviation). Correction is only performed when specific conditions are met (capable of accurately locating a single bit error), avoiding incorrect correction when the error cannot be determined or when multiple bits are present. Furthermore, the checksum adder 1221 allows selective checksum calculation of only critical bits based on the importance and type of data, balancing protection strength and system overhead. The protection linker 1223 tracks and records errors occurring at specific storage locations, providing a basis for system diagnostics and long-term reliability analysis.

[0086] In an embodiment of the present application, the third protection module 123 is configured to protect the computational integrity of the aforementioned systolic array 114 during the execution of the core matrix multiplication and addition operation through algorithmic fault-tolerant logic.

[0087] Directly integrating traditional (algorithmic fault-tolerant logic) ABFT algorithms into NPU hardware like the Gemmini is costly. This is primarily due to the specific requirements of the NPU's systolic array architecture. For example, the way data is read from the scratchpad is controlled by configuration registers (such as stride), and the size of the submatrices partitioned by the API and the amount of locally stored row data must be aligned with the number of PEs (processing units). Directly integrating ABFT would require significant modifications to the existing hardware architecture, software API, and task model.

[0088] To reduce adaptation costs, this application did not directly modify the core systolic array. Instead, it optimized the ABFT execution logic and designed a dedicated hardware unit—the Shield Group—to perform ABFT-related checksum calculations. This means that the core fault-tolerant calculations (checksum generation and verification) are outsourced to this dedicated hardware module, separate from but working in conjunction with the main computing array, rather than being deeply embedded in each PE of the systolic array or its control logic.

[0089] Therefore, in an embodiment of the present application, a shield array is provided within the third protection module 123. The shield array comprises a set of multipliers and multiple adder trees aligned with the number of processing units in each row of the systolic array. This design allows the shield array to handle checksum calculations associated with the main computational array. Furthermore, the shield array does not operate independently, but rather closely coordinates with the core matrix multiplication and addition process of the NPU's systolic array 114. Through specialized instructions and pipeline design, it preprocesses data, executes the core matrix multiplication and addition operations of the systolic array 114 in parallel, and performs post-calculation verification.

[0090] In the embodiment of the present application, before the core matrix multiplication and addition operation of the systolic array 114 begins or during the data loading phase, the data verifier 1224 and the data error corrector 1225 perform data verification and error correction on the input matrix involved in the operation. Furthermore, the checksum adder 1221 calculates an initial checksum vector corresponding to the input matrix according to the algorithmic fault-tolerant logic.

[0091] In an embodiment of the present application, when the systolic array 114 processes the loading of a specific matrix (e.g., a weight matrix), the shield array coordinates the transpose unit and the checksum adder 1221 in the nonlinear operator unit 115 to synchronously process the data flow to implement transposition and / or checksum calculation (e.g., generating a row checksum for the weight matrix).

[0092] In an embodiment of the present application, before the core matrix multiply-add operation of the systolic array 114 begins or during its data loading phase, the execution process of the systolic array 114 is overlapped with other operation phases of the systolic array 114 to hide latency.

[0093] In an embodiment of the present application, when the systolic array 114 performs its core matrix multiplication and addition operation to calculate a result matrix, the shield array calculates an expected checksum vector corresponding to the result matrix based on the initial checksum vector and the data input to the systolic array 114 according to the algorithmic fault-tolerant logic.

[0094] In an embodiment of the present application, during the aforementioned process of obtaining the expected checksum vector, the shield array first uses its multipliers and small adder trees aligned with the number of systolic array processing element rows to add column checksums to the input matrix A and row checksums to the weight matrix B using algorithmic fault-tolerant logic (ABFT). Next, the column checksums of the weight matrix are fixed in the shield array, and the input matrix A is streamed row by row, where each row is multiplied and accumulated with its corresponding checksum to calculate the column checksum of A×B. Similarly, the row checksum of A×B is calculated by fixing the row checksum of A in the shield array and streaming the transpose of B. The row and column checksums generated above are identical to those generated by traditional ABFT. Because the shield array utilizes a set of multipliers and several small adder trees aligned with the number of PEs per row, when a compute instruction is dispatched to the NPU, the system simultaneously generates a pre-computed sub-instruction. This sub-instruction enters the waiting queue along with the corresponding compute instruction, establishing an explicit dependency between preloading and computation.

[0095] Specifically, pre-calculation is performed first, triggering the data verifier 1224 and data error corrector 1225 to ensure the integrity of the original data. Then, the checksum adder 1221 calculates the checksum of A to obtain the initial checksum vector. Meanwhile, during the pre-loading process of B, the transposer and checksum adder 1221 receive the data stream and perform transposition and checksum calculations. At this stage, the shield array has acquired all necessary computational elements. Once the calculation is complete, the shield array performs matrix operations synchronously with the systolic array. Upon completion, the data verifier 1224 compares the results, while the data error corrector 1225 performs fault location, correction, and writeback operations. Finally, the A×B result is added to the bias matrix D in the accumulator to obtain the expected checksum vector.

[0096] Similarly, a corresponding adder is used in checksum adder 1221 to generate the final checksum. In this case, the final checksum is a checksum vector calculated based on the actual output of the systolic array. Specifically, when D is preloaded, the checksum adder generates a set of checksums for D. Following the same process, the calculated A×B checksum is added to the biased checksum to ensure that the entire calculation process is fully verified.

[0097] In one embodiment of the present application, the specific process of obtaining the A×B checksum by the shield array in the present application is compared with the specific process of obtaining the A×B checksum by the traditional ABFT. Figure 6 .

[0098] like Figure 6 As shown, first, the input matrix is Add the column checksum vector [4, 6] to A and add the column checksum vector [4, 6] as a new row to the bottom of A to get the matrix At the same time, the weight matrix Add the row checksum vector [7, 3] to B and add a new column of the row checksum vector [7, 3] to the right side of B to get the matrix .

[0099] In the process of obtaining the A×B checksum through the traditional ABFT, the matrix A CS With matrix B CS Multiplication: , the obtained column checksum vector of A×B is [13, 33], and the obtained row checksum vector of A×B is [28, 18].

[0100] When obtaining the checksum of A×B through the shield array, take the first row of A [1, 2] and the row checksum vector of B [7, 3] and perform a dot product operation: 1×7+2×3=7+6=13. This 13 is the column checksum of the first row of A×B. Take the second row of A [3, 4] and the row checksum vector of B [7, 3] and perform a dot product operation: 3×7+4×3=21+12=33. This 33 is the column checksum of the second row of A×B. At the same time, obtain the transpose of B. , take B T The first row of [4, 2] (i.e. the first column of B) is dot-producted with the column checksum vector of A [4, 6]: 4×4+2×6=16+12=28. This 28 is the row checksum of the first column of A×B. T The dot product of the second row [3, 1] of A (i.e., the second column of B) and the column checksum vector [4, 6] of A is: 3×4+1×6=12+6=18. This 18 is the row checksum of the second column of A×B.

[0101] In summary, obtaining an A×B checksum using the shield array is identical to obtaining an A×B checksum using traditional ABFT. This verifies the mathematical correctness of the shield array's operations, which allows it to calculate the expected checksum.

[0102] In the embodiment of the present application, after the pulsating array 114 generates the calculation result, the specific working process of the shield array can be referred to in Figure 7 .

[0103] Figure 7 An exemplary flow chart showing the specific working process of the shield array after the systolic array generates calculation results according to an embodiment of the present application is shown.

[0104] like Figure 7As shown, in step S710, the data verifier 1224 compares the expected checksum vector with the checksum vector calculated based on the actual output of the systolic array to determine whether there is a mismatch. In response to the absence of a mismatch, no action is performed in step S720. In response to the presence of a mismatch, in step S730, it is determined whether the mismatch satisfies a preset error correction condition. In response to the mismatch satisfying the preset error correction condition, in step S740, the shield triggers the aforementioned data error corrector 1225 to perform corresponding error correction. In response to the mismatch not satisfying the preset error correction condition, no action is performed.

[0105] In the embodiments of the present application, the preset error correction condition may be the existence of consistency deviation, or may be other error correction conditions set according to actual needs, and the present application does not limit this.

[0106] In an embodiment of the present application, when a continuous row or column mismatch is detected during a comparison between an expected checksum vector and a checksum vector calculated based on an actual output of a systolic array, a matrix position corresponding to the checksum vector where the mismatch first occurs in the continuous mismatch sequence is analyzed, and a fault processing unit within the systolic array that causes the continuous row or column mismatch is inferred based on the matrix position. The inferred fault processing unit is then marked in the error block.

[0107] In an embodiment of the present application, the area of local memory 113 where the mismatched data occurs, or the area of the tile of systolic array 114 where the mismatched data occurs, is updated based on the mismatch detected by data validator 1224. This links the error detection result (mismatch) with the physical or logical location where the error occurred. This allows fault tolerance to not only detect an error, but also determine where the error occurred. This precision makes subsequent error correction, isolation, or logging actions more efficient and targeted, avoids unnecessary global operations or large-scale checks, and reduces the overhead of the fault tolerance mechanism itself.

[0108] In an embodiment of the present application, a multi-stage pipeline architecture is employed during the operation of a systolic array, achieving overlapping execution of pipeline stages. Specifically, during the overlapping execution of pipeline stages achieved through the multi-stage pipeline architecture, the stage before the start of the core matrix multiplication-add operation of the systolic array is executed in time overlapping with the stage in which the systolic array performs its core matrix multiplication-add operation to calculate the result matrix. Furthermore, the stage after the systolic array generates the calculation result is executed in time overlapping with the stage in which the systolic array loads data.

[0109] In the embodiment of the present application, the multi-stage pipeline architecture can be found in Figure 8 .

[0110] Figure 8A schematic diagram of a multi-stage pipeline architecture according to an embodiment of the present application is shown.

[0111] like Figure 8 As shown, during the systolic array's first computation, before the core matrix multiplication and addition operation begins, the checksum adder calculates the checksum of the input matrix in stage #1-1. Simultaneously, during the preloading of the weight matrix, the transpose unit and checksum adder receive the data stream in stage #1-2 and perform transposition and checksum calculations. At this stage, the shield array has acquired all necessary computational elements. Once the computation is complete, the shield array executes the matrix operation synchronously with the systolic array in stage #1-3. Following completion, in stage #1-4, the data verifier compares the results, while the data error corrector performs fault location, correction, and writeback operations. The computational delays primarily occur during the verification, error correction, and checksum generation of the raw data (e.g., stages #1-1 and #1-2), and the verification and error correction of the post-computation (e.g., stage #1-4).

[0112] To hide these delays, the stages before the core matrix multiplication and addition operations of the systolic array begin are overlapped with the stages during which the systolic array performs its core matrix multiplication and addition operations to calculate the result matrix. The stages after the systolic array generates the result are overlapped with the data loading stages of the systolic array. For example, the first stage #2-1 of the second computation task (the stage before the core matrix multiplication and addition operations of the systolic array begin) is overlapped by stages #1-3 (the stage during which the systolic array performs its core matrix multiplication and addition operations to calculate the result matrix), and stages #1-4 (the stage after the systolic array generates the result) is overlapped by the second stage #2-2 of the second computation task (the data loading stage of the systolic array).

[0113] Through a multi-stage pipeline architecture, guard operators such as verification, error correction, and checksum calculations can be processed seamlessly with almost no slowdown. The resulting computational latency is effectively hidden, resulting in almost no additional performance overhead.

[0114] The third protection module 123 forms a protection mechanism for the NPU's core computing units (systolic arrays). By utilizing algorithmic fault-tolerant logic, errors that may occur during or after the systolic array performs core operations (such as matrix multiplication and addition) are detected, protecting the computation itself. Using the initial checksum vector of the input data and the ABFT logic, an expected checksum vector for the systolic array's computational results is predicted. This expected value is then compared with the checksum vector calculated based on the actual systolic array output to detect computational errors. When a detected checksum mismatch meets a preset condition (such as a consistency deviation), a data error corrector is triggered to correct the systolic array's computational results. By analyzing the pattern of checksum mismatches (particularly consecutive row / column mismatches), the specific processing unit within the systolic array that has failed can be inferred. Protection operations (such as input validation and checksum calculation) are overlapped with data loading or other operational phases of the systolic array to minimize the impact on the overall NPU performance. At the same time, other units (such as the transposer in the nonlinear operator unit) can be coordinated to work synchronously with the checksum adder to efficiently complete data preprocessing (such as weight transposition and checksum generation).

[0115] In an embodiment of the present application, the fourth protection module 124 is configured to verify the operation result of the nonlinear operator unit 115 based on the mathematical characteristics of the nonlinear operator executed by the nonlinear operator unit 115 or through hardware redundancy and output comparison mechanism.

[0116] In an embodiment of the present application, when the nonlinear operator executed by the nonlinear operator unit 115 belongs to the first type of nonlinear operator, the fourth protection module 124 verifies the operation result of the nonlinear operator unit based on the mathematical characteristics of the nonlinear operator executed by the nonlinear operator unit.

[0117] In an embodiment of the present application, the first type of nonlinear operator is a nonlinear operator with clear and easily verifiable mathematical constraint characteristics (such as LayerNorm, ReLU, Softmax).

[0118] Specifically, in the process of the fourth protection module 124 verifying the operation result of the nonlinear operator unit based on the mathematical characteristics of the nonlinear operator performed by the nonlinear operator unit, the fourth protection module 124 performs at least one of the following steps: checking whether the sum of all output values after LayerNorm normalization and offset is zero, whether the ReLU output is non-negative and whether the positive output position corresponds to the input position, and whether the Softmax output sum is one.

[0119] Since the output of LayerNorm should sum to 0 after normalization and offset processing, we use this property to verify its consistency. We verify its correctness by ensuring that there are no negative values in the ReLU output and that positive outputs correspond to the corresponding input positions. We also verify the consistency of Softmax based on the fact that the sum of Softmax outputs should be 1.

[0120] In an embodiment of the present application, when the nonlinear operator executed by the nonlinear operator unit 115 belongs to the second type of nonlinear operator, the fourth protection module 124 verifies the operation result of the nonlinear operator unit through hardware redundancy and output comparison mechanism.

[0121] In an embodiment of the present application, the second type of nonlinear operator is a nonlinear operator whose output mathematical characteristics are not obvious or whose rule-based verification logic implementation is too complex (such as GELU, Pooling).

[0122] During the process of the fourth protection module 124 verifying the operation result of the nonlinear operator unit through hardware redundancy and output comparison mechanism, the fourth protection module 124 retains multiple calculation copies of the corresponding nonlinear operator and compares the operation results generated by each of them one by one to detect errors in the operation process.

[0123] By choosing to adopt the fourth protection module 124, it is possible to balance the protection overhead and operator characteristics. Considering that the hardware cost of the nonlinear operator implemented by approximate calculation is already relatively low, a low-overhead characteristic verification method is preferentially adopted, and a redundant comparison method is only adopted when necessary (such as when the verification logic complexity is high), thereby providing customized reliability protection for different types of nonlinear operators in a resource-efficient manner.

[0124] In an embodiment of the present application, the control mechanism 130 triggers or manages the fault-tolerant operations of the plurality of hardware protection modules 120 by using an instruction set architecture including dedicated instructions through the aforementioned control unit 111 .

[0125] Specifically, the instruction set architecture (ISA), which includes dedicated instructions, provides a set of control interfaces that allow software to fine-tune and manage the hardware's protection features. These instructions cover configuration, triggering protection processes (memory and compute), performing error correction, and querying status and diagnostic information, demonstrating the tight integration of software and hardware within a full-stack architecture.

[0126] By employing control mechanism 130, the activation, operation, and information exchange of each protection module can be intelligently scheduled and managed, avoiding redundant detection and optimizing the protection process. Furthermore, by coordinating the protection modules, the performance overhead (e.g., latency) introduced by the protection mechanism itself can be minimized while ensuring adequate protection levels.

[0127] In summary, through the full-stack NPU system supporting a multi-level fault mitigation mechanism as provided above, the embodiments of the present application provide error detection and correction capabilities for at least one NPU functional block or data path within the NPU through multiple hardware protection modules. By coordinating the operation of multiple hardware protection modules through a control mechanism, computational errors within the NPU functional block or data path transmission errors caused by hardware failures, etc., can be promptly detected and corrected, preventing erroneous computational results from being used or output, and ensuring the accuracy of the final result. Even if partial hardware failure occurs, the error correction mechanism can largely mask the error, maintaining normal system operation, improving the overall system availability and task success rate, and thus significantly enhancing the reliability and robustness of the NPU. Furthermore, the use of multiple protection modules allows for the design of specialized, most effective protection mechanisms for different functional units or different types of errors within the NPU. Furthermore, protection modules can be flexibly deployed based on the critical paths and error-prone links of the NPU architecture, achieving focused protection of key NPU functional blocks and data paths while achieving broader coverage of the entire unit. Furthermore, the control mechanism intelligently schedules and manages the activation, operation, and information exchange of each protection module, avoiding redundant detection and optimizing the protection process.

[0128] Furthermore, in some embodiments, an ECC mechanism is employed in the first protection module to protect data stored in registers of the corresponding NPU functional blocks, using multiple odd parity bits to cover all relevant bits in the registers, thereby ensuring high register reliability. By configuring the ECC mechanism as a fully monitored ECC based on a multi-level parity tree, precise bit-level error detection and correction can be achieved using minimal redundant bits, significantly enhancing the reliability of both the control and storage paths.

[0129] Furthermore, in some embodiments, in the process of protecting the integrity of data in the local memory by a second protection module including a checksum adder, a protection register, a protection linker, a data verifier, and a data error corrector, the data verifier, in combination with the checksum adder and the protection register, can quickly detect whether there is an error when reading the data. After detecting an error, the data error corrector can use the checksum information to automatically repair errors within a certain degree to prevent erroneous data from being used. This reduces the risk of computing failure or system crash due to data corruption by automatically processing memory data errors at the hardware level, significantly improving the stable operation capability of the NPU in complex environments. Thus, a hardware-level, dedicated data integrity protection mechanism is provided for the local memory inside the NPU to ensure that the data stored therein does not undergo unexpected changes during reading, writing, and retention.

[0130] Furthermore, in some embodiments, a third protection module employs algorithmic fault-tolerant logic to protect the computational integrity of the systolic array during core matrix multiplication and addition operations. The shield array computes the expected checksum in parallel with the systolic array, rather than waiting for the final result before checking. This significantly reduces the performance latency associated with protection. The shield array utilizes hardware resources compatible with the systolic array, allowing for partial design reuse or leveraging similar processing units for checksum calculations. This reduces hardware overhead compared to fully redundantly replicating the entire systolic array. Input validation, error correction, and initial checksum calculations are performed during the data loading phase, utilizing pipelines to hide the latency of some protection operations. Error correction is initiated only when a mismatch meeting specific conditions is detected, avoiding unnecessary error correction overhead while enabling the correction of some computational errors and improving fault tolerance.

[0131] Furthermore, in some embodiments, the fourth protection module verifies the operation results of the nonlinear operator unit based on the mathematical characteristics of the nonlinear operator executed by the nonlinear operator unit or through hardware redundancy and output comparison mechanism. By combining the two strategies, the module can provide effective protection for different types of nonlinear operators, ensuring the overall reliability of the nonlinear processing link, and providing a set of differentiated, operator characteristic-based result verification mechanisms for the nonlinear operator units in the NPU, ensuring the correctness of these nonlinear transformation operation results that have an important impact on model accuracy.

[0132] Although multiple embodiments of the present application have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art can conceive of many changes, modifications, and alternatives without departing from the thought and spirit of the present application. It should be understood that in the process of practicing the present application, various alternatives to the embodiments of the present application described herein can be adopted. The accompanying claims are intended to define the scope of protection of the present application and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A full-stack NPU system supporting a multi-level fault mitigation mechanism, characterized in that: The system comprises: a plurality of NPU functional blocks for performing neural network computing tasks, the plurality of NPU functional blocks including at least: a control unit and an instruction reservation station for processing instructions, a local memory for storing data, a systolic array for performing core matrix operations, and a nonlinear operator unit for performing nonlinear operations; multiple hardware protection modules for providing error detection and correction capabilities for at least one NPU functional block or a data path within the NPU, the multiple hardware protection modules including a first protection module, a second protection module, a third protection module, and a fourth protection module, wherein the first protection module is configured to protect data stored in registers of the corresponding NPU functional block through an ECC mechanism, the second protection module is configured to protect the integrity of data in the local memory, the third protection module is configured to protect the computational integrity of the systolic array during core matrix multiplication and addition operations through algorithmic fault-tolerant logic, and the fourth protection module is configured to verify the operation results of the nonlinear operator unit based on the mathematical properties of the nonlinear operator executed by the nonlinear operator unit or through a hardware redundancy and output comparison mechanism; and A control mechanism to coordinate the operations of multiple hardware protection modules; The control mechanism uses an instruction set architecture including dedicated instructions to trigger or manage the fault-tolerant operations of the multiple hardware protection modules.

2. The full-stack NPU system supporting a multi-level fault mitigation mechanism according to claim 1, characterized in that: The ECC mechanism is a fully monitored ECC built on a multi-level parity tree; The ECC mechanism includes a plurality of partial odd parity bits and a total odd parity bit; The plurality of partial odd parity bits are generated by grouping the original data bits stored in the register, the grouping being based on a binary index of the original data bits themselves; Each part of the odd parity bits is used to monitor the set of original data bits whose corresponding bits in the binary index of the original data bits themselves are 1; The total odd parity bit is generated by aggregating results of the plurality of partial odd parity bits; When an error occurs in the data stored in the register of the corresponding NPU functional block, single error location and multi-bit error judgment are performed through the deviations corresponding to the partial odd parity bits and the deviation corresponding to the total odd parity bit.

3. The full-stack NPU system supporting a multi-level fault mitigation mechanism according to claim 2, characterized in that: In the process of protecting the data stored in the registers of the corresponding NPU functional block through the ECC mechanism, the following steps are performed: When an error occurs in the data stored in the register of the corresponding NPU functional block, it is determined whether deviations occur between each partial odd parity bit and the total odd parity bit; When some odd parity bits are deviated and all odd parity bits are deviated, single-bit error location is performed; When some odd parity bits are deviated but the total odd parity bits are not deviated, it is determined that multiple bit errors have occurred; as well as Error correction is performed based on the located error bits and odd parity check principle.

4. The full-stack NPU system supporting a multi-level fault mitigation mechanism according to claim 1, characterized in that: The second protection module includes: a checksum adder, configured to calculate one or more raw checksums based on the data to be written when the data is written to the local memory; a protection register, located in the local memory, for storing the original checksum; a protection linker, comprising a link block and an error block, wherein the link block is configured to work synchronously with the checksum adder when writing data, allocate a storage area for the data to be written, and establish an association between the location of the data to be written in the local memory and the storage location of its corresponding original checksum in the protection register; The error block is used to record the error occurrence history of the local memory and the specific location of the systolic array; a data validator configured to, when reading data from the local memory, recalculate one or more current checksums based on the read data, obtain, via the protection linker, an original checksum associated with the read data and stored in the protection register, and compare the recalculated current checksum with the obtained original checksum to detect whether there is a mismatch, wherein the mismatch is a numerical difference between the current checksum and the original checksum; and A data error corrector is configured to be activated when the data verifier detects the mismatch, and perform differentiated error processing based on the checksum difference information indicated by the detected mismatch.

5. The full-stack NPU system supporting a multi-level fault mitigation mechanism according to claim 4, characterized in that: During the differentiated error handling process performed by the data corrector, the following steps are performed: Determining whether the mismatch is that both the row checksum of a row and the column checksum of the corresponding column do not match; In response to the mismatch not being a mismatch between a row checksum of a row and a column checksum of a corresponding column, inhibiting activation of the data error corrector and not performing an error correction operation; In response to the mismatch being a mismatch between a row checksum of a row and a column checksum of a corresponding column, determining whether there is a consistency deviation; In response to the existence of the consistency deviation, the data error corrector corrects the single erroneous data bit by performing cross positioning to locate the single erroneous data bit using the consistency deviation value and updates the corresponding error record in the error block; In response to the absence of a consistency deviation, no action is performed.

6. The full-stack NPU system supporting a multi-level fault mitigation mechanism according to claim 4, characterized in that: In the process of the checksum adder generating the checksum for the data written into the local memory, the checksum adder selectively calculates the original checksum only for the key bits in the data word of the data to be written according to the data type and the preset sensitivity.

7. The full-stack NPU system supporting a multi-level fault mitigation mechanism according to any one of claims 4 to 6, characterized in that: The third protection module includes a shield array, wherein the shield array employs a plurality of adder trees and a set of multipliers aligned with the number of processing units per row of the systolic array; Wherein, during the operation of the systolic array, at least the following steps are performed: Before the core matrix multiplication and addition operation of the systolic array begins or during the data loading phase, the data verifier and the data error corrector perform data verification and error correction on the input matrix involved in the operation, and the checksum adder calculates an initial checksum vector corresponding to the input matrix according to the algorithmic fault-tolerant logic; When the systolic array performs its core matrix multiplication and addition operation to calculate the result matrix, the shield array calculates the expected checksum vector corresponding to the result matrix according to the algorithmic fault-tolerant logic using the initial checksum vector and the data input to the systolic array; When the systolic array generates a calculation result, the data verifier compares the expected checksum vector with the checksum vector calculated based on the actual output of the systolic array to determine whether there is a mismatch; In response to there not being a mismatch, performing no action; In response to a mismatch, determining whether the mismatch satisfies a preset error correction condition; In response to the mismatch satisfying a preset error correction condition, the shield triggers the data error corrector to perform corresponding error correction; In response to the mismatch not satisfying a preset error correction condition, no action is performed; wherein, when comparing the expected checksum vector with the checksum vector calculated based on the actual output of the systolic array, if a continuous row or column mismatch is detected, a matrix position corresponding to the checksum vector where the mismatch occurs for the first time in the continuous mismatch sequence is analyzed, and a fault processing unit within the systolic array that causes the continuous row or column mismatch is inferred based on the matrix position, and the fault processing unit is marked in the error block; Wherein, based on the mismatch detected by the data validator, an area in the local memory where the mismatched data is located or an area in the tile of the systolic array where the mismatched data is located is updated.

8. The full-stack NPU system supporting a multi-level fault mitigation mechanism according to claim 7, characterized in that: A multi-stage pipeline architecture is employed during operation of the systolic array, and overlapping execution of pipeline stages is achieved through the multi-stage pipeline architecture; In implementing overlapping execution of pipeline stages through the multi-stage pipeline architecture, the following steps are performed: overlapping in time the execution of a stage before the core matrix multiply-add operation of the systolic array begins with a stage during which the systolic array performs its core matrix multiply-add operation to calculate a result matrix; The stage after the systolic array generates a calculation result and the data loading stage of the systolic array are overlapped in time.

9. The full-stack NPU system supporting a multi-level fault mitigation mechanism according to claim 1, wherein: When the nonlinear operator executed by the nonlinear operator unit belongs to the first type of nonlinear operator, the fourth protection module verifies the operation result of the nonlinear operator unit based on the mathematical characteristics of the nonlinear operator executed by the nonlinear operator unit. In the process of the fourth protection module verifying the operation result of the nonlinear operator unit based on the mathematical characteristics of the nonlinear operator executed by the nonlinear operator unit, the fourth protection module performs at least one of the following steps: checking whether the sum of all output values after LayerNorm normalization and offset is zero, whether the ReLU output is non-negative and whether the positive output position corresponds to the input position, and whether the Softmax output sum is one; When the nonlinear operator executed by the nonlinear operator unit belongs to the second type of nonlinear operator, the fourth protection module verifies the operation result of the nonlinear operator unit through hardware redundancy and output comparison mechanism. In the process of the fourth protection module verifying the operation result of the nonlinear operator unit through hardware redundancy and output comparison mechanism, the fourth protection module retains multiple calculation copies of the corresponding nonlinear operator and compares the operation results generated by each of them one by one to detect errors in the operation process.

Images