A neural network hardware acceleration apparatus and method

Abstract

The application discloses a neural network hardware acceleration device and method, and the device comprises a neural network hardware accelerator, wherein a GEMM / ALU microinstruction buffer is connected to a calculation module; a control module analyzes machine instructions and distributes microinstructions to the microinstruction buffer; a load module loads data from an external storage to an input / weight buffer; in the calculation module, a GEMM control unit and an ALU control unit control the access to the corresponding microinstruction buffer respectively; a GEMM calculation core, an ALU calculation core and a pipeline register connected to the two calculation cores; for a pure ALU operator, only the ALU calculation core is enabled, data is directly obtained from the input / weight buffer, and the output of the calculation core is connected to an output buffer; for a composite operator comprising GEMM and ALU operations, the two calculation cores are used in cooperation to transmit data through the pipeline register; and a store module stores the data in the output buffer to an external global storage.

Description

Technical Field

[0001] This invention relates to the field of neural network hardware acceleration technology, specifically to a neural network hardware acceleration device and method based on GEMM-ALU operation fusion processing. Background Technology

[0002] The computation section of existing neural network hardware accelerators typically comprises two main modules: General Matrix Multiply (GEMM) and Arithmetic Logic Unit (ALU). The core function of the GEMM module is to perform large-scale, intensive matrix multiplication operations in core neural network operators such as convolutional layers, fully connected layers, and attention mechanisms, pursuing ultimate parallel computing efficiency and throughput. The core function of the ALU module is to handle various auxiliary, element-level arithmetic and logical operations surrounding the GEMM, ensuring the complete execution of the entire neural network computation graph. The ALU module is usually more flexible than the GEMM, capable of handling a wider variety of operation types. Typical ALU operations include, but are not limited to: 1) Activation functions: calculation of linear functions such as ReLU and Sigmoid; 2) Quantization / dequantization: such as the process of quantizing the accumulated value from 32 bits to 8 bits in the tflite INT8 quantization specification; 3) Data type conversion: limiting the 8-bit output to the range [-128, 127] through MIN and MAX operations. The computational process of a neural network layer is typically: data preparation -> GEMM core computation -> ALU operation processing -> data output. That is, the GEMM and ALU modules need to work together to complete the inference task of the neural network and maximize the performance of the hardware accelerator.

[0003] Existing neural network processors typically employ an independent parallel or serial architecture for the GEMM and ALU modules. Data transfer between the GEMM and ALU modules occurs in two ways: 1) Transfer via an accumulation buffer: For example, VTA's GEMM and ALU modules use an independent serial architecture, both belonging to the COMPUTE level. The GEMM module uses inner product calculation, with two modes: reset and accumulation. The ALU module executes in blocks of data, and each ALU instruction supports only one type of ALU operation. Panacea's GEMM and ALU modules use an independent parallel architecture. The GEMM module uses outer product calculation, and the ALU module uses a parallel pipeline architecture, supporting a relatively fixed and limited sequence of ALU operations. Gemmini designs a hardware-selectable complex instruction, LoopConv and LoopMatMul, to achieve more efficient scheduling of basic instructions Mvin, PreLoad, Compute, and Mvout through hardware. Mvout supports parallel execution of three ALU operations: Normalization, Activation, and MaxPolling. GEMM supports WS and OS data streams, and data transfer between GEMM and ALU occurs via an on-chip buffer. 2) Data transfer via registers: For example, the ALU module can receive data online from the GEMM module or from a local buffer. The GEMM module uses inner product calculation, while the ALU module uses a parallel pipeline architecture, executing on a single data element. The supported ALU operation sequences are relatively fixed and limited. The former (GEMM and ALU instructions) is simpler and clearer in software programming, and offers more flexible upper-level data flow support. Its disadvantages are a larger accumulation buffer area and higher read / write power consumption. The latter (GEMM and ALU instructions) integrates GEMM and ALU instructions into a single, more complex instruction at the software level. Its advantage is that it doesn't require an accumulation buffer, resulting in higher energy efficiency. Its disadvantages are reliance on the underlying hardware to handle fine-grained dependencies between GEMM and ALU, limited data flow support, and the need to prioritize accumulation calculations.

[0004] Furthermore, neural network operators have varying requirements for ALU operation sequences. On one hand, users may adjust the quantization method to balance model accuracy and inference speed. For example, the tflite INT8 quantization specification requires the implementation of the RMUL+SHIFT+ADD ZeroPoint sequence, while the Global quantization method used in the TVM compiler only requires the SHIFT sequence. On the other hand, users expect hardware accelerators to support not only core operators such as convolution and fully connected layers, but also residual add (ResAdd) operators. Residual add requires the implementation of dequantization + addition + quantization operations, further exacerbating the differences in ALU operation sequences.

[0005] Therefore, there is an urgent need for a neural network hardware acceleration device and method that balances flexibility and high energy efficiency, supporting both independent parallel processing of GEMM-ALU and meeting the differentiated ALU operation sequence requirements, so as to support the implementation of neural network inference applications in power- and resource-sensitive terminal devices. Summary of the Invention

[0006] To address the aforementioned technical problems in the existing technology, this invention proposes a hardware acceleration method and apparatus for neural networks based on GEMM-ALU operation fusion, the specific technical solution of which is as follows:

[0007] A neural network hardware acceleration device includes a host CPU, external global storage, and a neural network hardware accelerator connected to each other. The neural network hardware accelerator includes a control module, a load module, a computing module, and a store module.

[0008] The control module is responsible for receiving and decoding machine instructions and generating micro-instruction control signals.

[0009] The Load module is configured to load data from external global storage into the input buffer and the weight buffer;

[0010] The Store module is configured to store the output buffer data in external global storage;

[0011] The computing module is connected to a GEMM micro-instruction buffer and an ALU micro-instruction buffer, which are used to store micro-instructions loaded from external global storage, respectively.

[0012] The computing module integrates:

[0013] The GEMM control unit and the ALU control unit are used to control access to the GEMM micro-instruction buffer and the ALU micro-instruction buffer, respectively.

[0014] It employs an independent parallel architecture for the GEMM computing core and the ALU computing core, as well as a pipelined register connecting the GEMM computing core and the ALU computing core;

[0015] The ALU computing core adopts a single instruction multiple data architecture, supports ALU operation fusion processing function with one-dimensional tensors as the unit, and the output is connected to the output buffer.

[0016] Furthermore, the GEMM computing core, pipeline register, and ALU computing core are synchronized using a Valid, Ready handshake mechanism, and data transmission only occurs when both Valid and Ready signals are valid simultaneously.

[0017] Furthermore, the Load module, the Computing module, and the Store module are connected to form a pipeline architecture and synchronized through a synchronization queue.

[0018] Furthermore, when the computing module executes a pure ALU operator, only the ALU control unit and the ALU computing core are working, and the data of the ALU computing core comes directly from the input buffer and the weight buffer; when executing a composite operator that includes GEMM and ALU operations, all components in the computing module except the ALU control unit are working, and the GEMM control unit sends control signals to the GEMM computing core and the ALU computing core respectively to realize the fusion processing of GEMM and ALU operations.

[0019] Furthermore, the machine instructions include the following fields:

[0020] Opcode: Specifies the task type, including core operators such as convolutional / fully connected layers for GEMM and ALU operation tasks;

[0021] Dependency signals: used to handle write-after-read and read-after-write dependencies between the Load unit, ALU computing core, and Store unit;

[0022] Loop iteration count: including iter_out, iter_in and iter_ws, where iter_out and iter_in define the number of iterations of the main loop inside and outside the ALU instruction, and iter_ws is implicitly included in the micro-instruction loop of GEMM, defining the number of times the GEMM micro-instruction performs the cumulative calculation, so that the fusion processing of GEMM and ALU operations adds weighted reuse under output multiplexing;

[0023] Microinstruction pointer: Defines the start and end pointers of the GEMM microinstruction sequence required by the machine instruction in the GEMM microinstruction buffer, and defines the start and end pointers of the ALU microinstruction sequence required by the machine instruction in the ALU microinstruction buffer;

[0024] Loop address factor: Defines two address offset factors configured by the software in the main loop inside and outside the machine instruction;

[0025] Loop dimension labels: Define the dimension labels for mapping the inner and outer main loops.

[0026] Furthermore, the machine instructions employ a dimension-dependent compression encoding method, which utilizes the correlation between dimensions and input, weight, output, and bias data to compress the number of cyclic address factors.

[0027] The ALU microinstructions define the ALU operation type, operand register index, operand value, or address offset.

[0028] Furthermore, the GEMM computing core includes:

[0029] Multiple multiply-accumulate units employ an inner product calculation mode: accumulation calculation is performed within each multiply-accumulate unit, while parallel calculation is performed between multiply-accumulate units.

[0030] A set of weight registers;

[0031] Multiple sets of accumulator registers, the number of which can be configured by hardware, determine the maximum number of times the weights can be reused;

[0032] The GEMM computing core actively triggers the transmission of the accumulated result to the pipeline register when the accumulated calculation is completed, while the ALU computing core actively triggers the reading of the accumulated result to the pipeline register and stores the final calculation result in the output buffer when its ALU operation sequence is completed.

[0033] The accumulator register and the pipeline register are mapped one-to-one, and the data at the accumulator register entry n will only be transmitted to the pipeline register entry n.

[0034] Furthermore, the ALU computing core includes multiple ALU registers and a selector. The selector selects data from the pipeline registers and loads it into the corresponding ALU registers based on the information from the ALU microinstructions.

[0035] Furthermore, a dimension-mapping-based scheduling optimization method is adopted to provide register-level and buffer-level reuse of input, weight, and accumulated value data.

[0036] A neural network acceleration method includes:

[0037] The Load module loads data from external global storage into the input buffer and weight buffer.

[0038] The control module parses machine instructions and distributes microinstructions to the GEMM microinstruction buffer and the ALU microinstruction buffer.

[0039] Select the working mode based on the operator type:

[0040] For pure ALU operators, only the ALU computation core is enabled, and data is obtained directly from the input / weight buffer.

[0041] For composite operators that include GEMM and ALU operations, the GEMM computing core and the ALU computing core are used in concert, and data is transferred through pipelined registers;

[0042] The ALU calculates the core output and sends the final result to the output buffer.

[0043] The Store module writes the final result from the output buffer back to external global storage.

[0044] Beneficial effects: This invention can effectively utilize instruction fields, avoid the use of accumulation buffers, and effectively reduce the area of ​​hardware accelerators; combined with the ALU operation fusion processing module and the data flow of WS first and OS later, it can improve the energy efficiency of hardware accelerators while maintaining flexibility. Attached Figure Description

[0045] Figure 1 is a block diagram of the structural principle of the neural network hardware acceleration device based on GEMM-ALU operation fusion processing of the present invention;

[0046] Figure 2 is a pseudocode diagram of the instruction encoding method and CISC instruction set for GEMM-ALU operation fusion processing according to the present invention;

[0047] Figure 3 is a hardware microarchitecture diagram of the GEMM-ALU operation fusion processing that supports WS-OS data multiplexing according to the present invention.

[0048] Figure 4 is a pseudocode diagram of the scheduling optimization method based on GEMM-ALU operation fusion processing of the present invention;

[0049] Figure 5 is a schematic diagram of data transmission between the GEMM computing core and the pipeline register of the present invention.

[0050] Figure 6 is a schematic diagram of data transmission between the pipeline register and the ALU computing core of the present invention;

[0051] Figure 7 This is a schematic diagram of the convolution calculation method of the existing VTA and the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0053] like Figure 1As shown in this embodiment, a neural network hardware acceleration device based on GEMM-ALU operation fusion includes a host CPU, external global storage, an on-chip bus interface, and a neural network hardware accelerator. The neural network hardware accelerator includes a control module, a load module, a computation module, and a store module. The control module includes registers, an instruction queue (INS_QUEUE), and instruction fetch / decode logic. The load module is responsible for loading data from external global storage into the input buffer (inp_buff) and the weight buffer (wgt_buff). The store module is responsible for storing data from the output buffer (out_buff) into external global storage. The computation module is the core of this invention, responsible for loading microinstructions from external global storage into the GEMM microinstruction buffer (GEMM Uop Buffer (guop_buff)) and the ALU microinstruction buffer (ALU Uop Buffer (auop_buff)). The computation module integrates a GEMM control unit, an ALU control unit, a GEMM computation core, an ALU computation core, and pipelined registers connecting the GEMM computation core and the ALU computation core. Because the GEMM and ALU computing cores employ an independent and parallel architecture, the computing module requires two independent microinstruction buffers: the GEMM microinstruction buffer and the ALU microinstruction buffer. Access to the GEMM and ALU microinstruction buffers is sent by the GEMM control unit and the ALU control unit, respectively. The ALU computing core supports ALU operation fusion processing functions based on one-dimensional tensors. Microinstructions define the specific ALU operations and functions to be executed, and the one-dimensional tensor represents multiple data elements; that is, the ALU computing core adopts a Single Instruction Multiple Data (SIMD) architecture. When executing pure ALU operators such as ResAdd, only the ALU control unit and the ALU computing core are operational in the computing module. The data for the ALU computing core comes from the input buffer and weight buffer, not the pipeline registers. When executing operators involving both GEMM and ALU operations, such as convolution and fully connected layers, all components in the computing module except the ALU control unit are active. The GEMM control unit sends corresponding control signals to both the GEMM and ALU computing cores to achieve fusion processing of GEMM and ALU operations. The GEMM computing core, pipeline registers, and ALU computing core use a simple Valid, Ready handshake mechanism for synchronization. Data transfer between the upper and lower levels is successful only when both Valid and Ready are pulled high simultaneously. The GEMM computing core actively triggers the transfer of the accumulation result to the pipeline register when the accumulation calculation is completed.The ALU computing core actively triggers the reading of the accumulated result from the pipeline register when its ALU operation sequence is completed. Only the ALU computing core in the computing module can store the final result in the output buffer. The Load module, computing module, and Store module adopt a pipelined architecture and are synchronized through a synchronization queue to avoid read-after-write (RAW) and write-after-read (WAR) conflicts in the local buffer.

[0054] like Figure 2 As shown, an instruction encoding method and CISC instruction set for GEMM-ALU fusion processing are designed, which includes two parts: user / software visible machine instructions and low-level microinstructions responsible for fine control of CISC instruction execution.

[0055] The machine instruction has a bit width of 128 bits, and its key fields include opcode, dependent signals, loop iteration count, microinstruction pointer, loop address factor, and loop dimension label, as detailed below:

[0056] Opcode: Composed of opcodes, it specifies the task type and supports core operators such as convolutional / fully connected layers that include GEMM and ALU operation tasks. It does not support depthwise convolution.

[0057] Dependency signals (dep_flags): Composed of pop_prev_dep, pop_next_dep, push_prev_dep, and push_next_dep, they are responsible for handling write-after-read (WAR) and read-after-write (RAW) dependencies between the Load module, the computation module, and the Store module, enabling the software to better maintain the local buffers in the neural network processor.

[0058] The loop iteration count consists of iter_out, iter_in, and iter_ws. iter_out and iter_in define the number of iterations in the main loop inside and outside the ALU instruction, while iter_ws is implicitly included in the GEMM microinstruction loop. That is, the number of GEMM microinstructions for this instruction is a multiple of iter_ws, representing the number of accumulation calculations performed through GEMM microinstructions. iter_ws is a key component of this invention. Register-based GEMM-ALU operation fusion processing generally uses output stationary (OS) to control register overhead. Adding a certain amount of weight stationary (WS) under the OS can significantly reduce the amount of local buffer access for weights, further improving the energy efficiency of the neural network processor.

[0059] Microinstruction pointers: Composed of guop_bgn, guop_end, auop_bgn, and auop_end, they define the start pointer (guop_bgn) and end pointer (guop_end) of the GEMM microinstruction sequence required by the machine instruction in the GEMM microinstruction buffer, and the start pointer (auop_bgn) and end pointer (auop_end) of the ALU microinstruction sequence required by the machine instruction in the ALU microinstruction buffer.

[0060] Loop address factors: Composed of factor_out0, factor_in0, factor_out1, and factor_in1, these factors define two address offset factors configured by the software in the main loop inside and outside the machine instruction. They are not explicitly assigned to output, input, or weighted data blocks and belong to a compressed encoding format.

[0061] Loop dimension labels: Composed of dim_out and dim_in, these labels define the dimension labels for mapping the inner and outer main loops. Optional dimensions are {DIM_OC, DIM_OH, DIM_OW, DIM_KC, DIM_KH, DIM_KW}.

[0062] The underlying microinstructions consist of two parts: GEMM microinstructions and ALU microinstructions. GEMM microinstructions include address offsets for outputs, inputs, and weights. The ALU operation fusion processing module is characterized by implementing ALU operation fusion processing through ALU microinstructions. These ALU microinstructions define the ALU operation type, operand register index, operand value or address offset, and other necessary fields.

[0063] The instruction encoding method based on dimensional correlation is one of the key contents of this invention. GEMM-ALU operation fusion processing requires providing the address offsets and address offset factors of the input, weights, biases, and output data blocks, as well as the address pointers of GEMM and ALU microinstructions. How to compress the information of two instructions into one instruction is a challenging problem.

[0064] The instruction encoding method of this invention utilizes the correlation between dimension and input, weight, output, and bias data. It was found that providing loop address factors based on data has some redundancy; therefore, two loop address factors are provided for each loop. For specific encoding methods, see [link to specific instructions]. Figure 2The `get_factor` function in the instruction encoding. For standard convolution, dimensions DIM_KH, DIM_KW, and DIM_KC are related to the input and weights, but not to the output; dimensions DIM_OH and DIM_OW are related to the input and output, but not to the weights; and dimension DIM_OC is related to the output and weights, but not to the input. Using this instruction encoding method, the original six required loop address factors: `dst_factor_out`, `dst_factor_in`, `src_factor_out`, `src_factor_in`, `wgt_factor_out`, and `wgt_factor_in` can be compressed into four loop address factors: `factor_out0`, `factor_in0`, `factor_out1`, and `factor_in1`. Furthermore, considering that the operands required in the ALU operation sequence, such as biases and quantization parameters, are usually scalars or one-dimensional vectors, and that one-dimensional vectors are related to DIM_OC, the instruction encoding does not directly provide the address factors for operands such as biases, but instead uses implicit derivation, as detailed in [see...]. Figure 2 The `get_bias_idx` function in the code. When the attribute of the inner and outer main loops is `DIM_OC`, the index of that loop is the `bias_idx` value; otherwise, the `bias_idx` value is 0. The former corresponds to a one-dimensional vector, and the latter corresponds to a scalar. It's important to note that this encoding method requires that the attribute of the inner and outer main loops cannot both be `DIM_OC` simultaneously; otherwise, two address factors, similar to input, weight, and output, would need to be provided for the bias, which is quite costly. This assumption is reasonable and feasible because the field widths of `iter_out` and `iter_in` are very large, eliminating the need to split them into two loops.

[0065] The pseudocode for GEMM-ALU arithmetic fusion instructions is as follows: Figure 2 As shown in the lower left corner, the GEMM controller manages the inner and outer main loops, handles the compression and decoding of instruction encodings such as get_factor and get_bias_idx, and manages the synchronous execution of the GEMM computing core and the ALU computing core. It can be seen that the pseudocode contains two main code segments. The upper part is the GEMM section, containing acc_reg and idx_reg to store intermediate accumulation results and output addresses. The activation index acc_idx is derived from guop_idx, guop_bgn, and iter_ws. The GEMM micro-instruction loop implicitly executes iter_ws first, followed by the accumulation calculation of acc_reg. The lower part is the ALU section, containing explicit iter_ws and the ALU micro-instruction loop. The intermediate output results of the ALU micro-instructions are stored in the ALU registers. After the ALU micro-instruction sequence is completed, the register results are stored in the corresponding position in the output buffer according to the address of idx_reg.

[0066] like Figure 3 As shown, a GEMM-ALU arithmetic fusion processing hardware microarchitecture supporting WS-OS data multiplexing is designed. The GEMM computing core contains multiple sets of Multiply Accumulation (MAC) units, a set of weight registers, and multiple sets of accumulation registers. The MAC units employ an inner product calculation mode, meaning accumulation calculations are performed within each MAC unit, while parallel calculations are performed between MAC units. Therefore, input data is transmitted to the MAC units via multicast, while weight data is transmitted to the MAC units in parallel. Each MAC unit outputs only one intermediate accumulation result, and the combination of intermediate accumulation results from multiple MACs corresponds to one entry in the accumulation register. The number of entries in the accumulation registers can be configured by the hardware, and its size determines the maximum number of weight reuses that the hardware can support. The accumulation registers and pipeline registers use a one-to-one mapping, meaning that data at entry n of the accumulation register will only be transmitted to entry n of the pipeline register. Once the accumulation calculation at a certain accumulator register entry is completed, the register will pull the Valid signal high, preparing the output address and accumulation result. If the corresponding pipeline register entry is in the Ready state, it means that the data in the accumulator register can be transferred to the pipeline register; otherwise, the data in the accumulator register will remain. If the data in the accumulator register cannot be transferred, the GEMM calculation core will be stalled until the accumulator register becomes idle.

[0067] The ALU core can only execute one ALU operation sequence at a time, based on the data entering the pipelined register. When the ALU operation sequence is incomplete, the ALU core pulls the Ready signal low, indicating that it cannot receive data transmission from the pipelined register. The ALU core contains a selector, whose function is to select data from the pipelined register based on information from the ALU microinstructions and load it into the corresponding ALU register. The ALU core supports fused processing of operations such as ADD, RMUL, SHIFT, MIN, and MAX. When the ALU operation sequence is complete, the ALU core writes the result to the output buffer and pulls the Ready signal high, ready to receive new data.

[0068] like Figure 4 As shown, a scheduling optimization method based on GEMM-ALU operation fusion processing is designed, which has the following characteristics:

[0069] Chunk Size: Considering the symmetry between the two dimensions DIM_OH and DIM_OW, the chunk size of the DIM_OW dimension is larger by default than that of the DIM_OH dimension. Since the DIM_OH dimension needs to be mapped to iter_ws, the chunk size of the DIM_OH dimension will be limited by the number of entries in the accumulator register. For scenarios where DIM_OH is larger than DIM_OW, the roles of DIM_OH and DIM_OW can be swapped, and the DIM_OW dimension can be mapped to iter_ws.

[0070] Dimension mapping: in Figure 4 The pseudocode for a typical scenario is as follows: the DIM_OH dimension is mapped to iter_ws to provide register-level weight data reuse; the DIM_KH, DIM_KW, and DIM_KC dimensions are mapped to the GEMM microinstruction loop to provide register-level intermediate accumulated value data reuse; the DIM_OH and DIM_OW dimensions are mapped to the inner loop to provide buffer-level weight data reuse; and the DIM_OC dimension is mapped to the outer loop to provide buffer-level input data reuse. This dimension mapping method comprehensively considers both register-level and buffer-level input, weight, and accumulated value data reuse, thus fully optimizing the memory access power consumption of the neural network processor. For scenarios where DIM_KH*DIM_KW*DIM_KC is large or DIM_OH / DIM_OW / DIM_OC=1, the DIM_KH, DIM_KW, and DIM_KC dimensions can be mapped to the inner and outer main loops. (Continue to refer to...) Figure 4 However, the dimensions DIM_KH, DIM_KW, and DIM_KC must be mapped to consecutive loops, without any loops mapped to DIM_OH, DIM_OW, or DIM_OC in between. Furthermore, the loops mapped to the dimensions DIM_KH, DIM_KW, and DIM_KC must be more internal than those mapped to DIM_OH, DIM_OW, and DIM_OC. This characteristic is due to the GEMM-ALU operation fusion processing and is limited by a finite number of accumulation registers.

[0071] Reference Figure 5 This illustrates the data transfer process between the GEMM computing core and the pipeline registers. In the diagram, acc_start represents the first accumulation calculation in the acc_reg register, corresponding to... Figure 2 The initial `iter_ws` GEMM microinstructions in the pseudocode execute, functionally clearing `acc_reg` and then accumulating the GEMM calculation result. `acc_done` indicates the last accumulation calculation of the accumulator register, corresponding to... Figure 2The last `iter_ws` GEMM microinstructions in the pseudocode are executed. After `acc_reg[i]` finishes accumulating, its data will be transferred to the pipeline register `pipeline_reg`, triggering an update of the pipeline register. It can be seen that the data transfer between the accumulator register and the pipeline register is based on the entry point, and they are independent of each other, traversing in a fixed ascending order.

[0072] Reference Figure 6 This diagram illustrates the data transfer process between the pipeline register and the ALU compute core. After being updated by the GEMM compute core, the pipeline register raises its Valid value high. When alu_ready is high, the pipeline register sends an entry data to the ALU compute core. Simultaneously, the Valid value of this pipeline register's entry point goes low in the next cycle, indicating that the data transfer has been successful and the internal data is no longer valid. alu_ready goes high once every auop_end-auop_bgn cycles, indicating that the ALU compute core is executing the ALU microinstruction sequence.

[0073] from Figure 5 and Figure 6 As can be seen, in the case of GEMM-ALU fusion processing, the GEMM computing core, due to output multiplexing, continuously outputs the accumulated result for iter_ws cycles, while the ALU computing core can only process one accumulated result every auop_end-auop_bgn cycles. This results in a mismatch in instantaneous processing bandwidth, a problem addressed by the pipeline register design. Whether the long-term processing bandwidth matches depends on the computational intensity of the target application. Taking convolution as an example, if DIM_KH * DIM_KW * DIM_KC is greater than auop_end-auop_bgn, then the ALU's processing time can be completely hidden within the GEMM's processing time, and the overall performance is determined by the GEMM. Otherwise, the overall performance is determined by the ALU.

[0074] Reference Figure 7Figure a) illustrates the convolution computation method of this invention compared to existing VTA accelerators. The VTA's GEMM and ALU modules are logically independent but belong to the same computation stage and are executed serially. Furthermore, the ALU module uses data blocks as the computation granularity, and each ALU instruction can only perform one ALU operation. Data transfer between the GEMM and ALU modules is achieved through an accumulation buffer; since they are executed sequentially, no synchronization signal is needed. In the VTA convolution implementation, GEMM computation depends on the input and weights, ALU computation depends on the GEMM output and bias, and output storage depends on ALU computation. Figure a) shows a schematic diagram of the computation process using the PING-PONG execution method. It can be seen that this method involves very frequent access to the accumulation buffer, and the bandwidth and read / write ports of the accumulation buffer become performance bottlenecks. In this invention, the GEMM and ALU modules have independent logic; although they belong to the same computation stage, they are executed in parallel. The ALU module supports vector-based ALU operation fusion, which is smaller in granularity than data blocks. The GEMM and ALU modules transmit data through pipelined registers, avoiding the use of the accumulation buffer. The method for implementing convolution calculation in this invention is shown in Figure b). Since GEMM and ALU operations are integrated, the input, weights, and biases need to be prepared beforehand. Once the GEMM has completed its accumulated output, the ALU module can begin computation, exhibiting a distributed characteristic. As can be seen from the figure, the ALU's computation time can be effectively hidden within the GEMM's computation process, significantly improving the performance of the neural network processor.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A neural network hardware acceleration device, comprising a host CPU, external global storage, and a neural network hardware accelerator interconnected, characterized in that: The neural network hardware accelerator includes a control module, a load module, a computing module, and a store module; The control module is responsible for receiving and decoding machine instructions and generating micro-instruction control signals. The Load module is configured to load data from external global storage into the input buffer and the weight buffer; The Store module is configured to store the output buffer data in external global storage; The computing module is connected to a GEMM micro-instruction buffer and an ALU micro-instruction buffer, which are used to store micro-instructions loaded from external global storage, respectively. The computing module integrates: The GEMM control unit and the ALU control unit are used to control access to the GEMM micro-instruction buffer and the ALU micro-instruction buffer, respectively. It employs an independent parallel architecture for the GEMM computing core and the ALU computing core, as well as a pipelined register connecting the GEMM computing core and the ALU computing core; The ALU computing core adopts a single instruction multiple data architecture, supports ALU operation fusion processing function with one-dimensional tensors as the unit, and the output is connected to the output buffer.

2. The apparatus according to claim 1, characterized in that, The GEMM computing core, pipeline register, and ALU computing core are synchronized using a Valid and Ready handshake mechanism, and data transmission only occurs when both Valid and Ready signals are valid.

3. The apparatus according to claim 1, characterized in that, The Load module, Compute module, and Store module are connected to form a pipeline architecture and synchronized through a synchronization queue.

4. The apparatus according to claim 1, characterized in that, When the computing module executes a pure ALU operator, only the ALU control unit and the ALU computing core are working. The data of the ALU computing core comes directly from the input buffer and the weight buffer. When executing a composite operator that includes GEMM and ALU operations, all components in the computing module except the ALU control unit are working. The GEMM control unit sends control signals to the GEMM computing core and the ALU computing core respectively to realize the fusion processing of GEMM and ALU operations.

5. The apparatus according to claim 4, characterized in that, The machine instructions include the following fields: Opcode: Specifies the task type, including the core operators of convolutional / fully connected layers for GEMM and ALU operation tasks; Dependency signals: used to handle write-after-read and read-after-write dependencies between the Load module, the ALU computing core, and the Store module; Loop iteration count: including iter_out, iter_in and iter_ws, where iter_out and iter_in define the number of iterations of the main loop inside and outside the ALU instruction, and iter_ws is implicitly included in the micro-instruction loop of GEMM, defining the number of times the GEMM micro-instruction performs the cumulative calculation, so that the fusion processing of GEMM and ALU operations adds weighted reuse under output multiplexing; Microinstruction pointer: Defines the start and end pointers of the GEMM microinstruction sequence required by the machine instruction in the GEMM microinstruction buffer, and defines the start and end pointers of the ALU microinstruction sequence required by the machine instruction in the ALU microinstruction buffer; Loop address factor: Defines two address offset factors configured by the software in the main loop inside and outside the machine instruction; Loop dimension labels: Define the dimension labels for mapping the inner and outer main loops.

6. The apparatus according to claim 5, characterized in that, The machine instructions employ a dimension-dependent compression encoding method, which utilizes the correlation between dimensions and input, weight, output, and bias data to compress the number of cyclic address factors; the ALU microinstructions define the ALU operation type, operand register index, operand value, or address offset.

7. The apparatus according to claim 1, characterized in that, The GEMM computing core includes: Multiple multiply-accumulate units employ an inner product calculation mode: accumulation calculation is performed within each multiply-accumulate unit, while parallel calculation is performed between multiply-accumulate units. A set of weight registers; Multiple sets of accumulator registers, the number of which can be configured by hardware, determine the maximum number of times the weights can be reused; The GEMM computing core actively triggers the transmission of the accumulated result to the pipeline register when the accumulated calculation is completed, while the ALU computing core actively triggers the reading of the accumulated result to the pipeline register and stores the final calculation result in the output buffer when its ALU operation sequence is completed. The accumulator register and the pipeline register are mapped one-to-one, and the data at the accumulator register entry n will only be transmitted to the pipeline register entry n.

8. The apparatus according to claim 1, characterized in that, The ALU computing core includes multiple ALU registers and a selector. The selector selects data from the pipeline registers and loads it into the corresponding ALU registers based on the information from the ALU microinstructions.

9. The apparatus according to claim 1, characterized in that, A scheduling optimization method based on dimension mapping is adopted to provide register-level and buffer-level reuse of input, weight, and accumulated value data.

10. A neural network acceleration method using the apparatus according to any one of claims 1-9, characterized in that, include: The Load module loads data from external global storage into the input buffer and weight buffer. The control module parses machine instructions and distributes microinstructions to the GEMM microinstruction buffer and the ALU microinstruction buffer. Select the working mode based on the operator type: For pure ALU operators, only the ALU computation core is enabled, and data is obtained directly from the input / weight buffer. For composite operators that include GEMM and ALU operations, the GEMM computing core and the ALU computing core are used in concert, and data is transferred through pipelined registers; The ALU calculates the core output and sends the final result to the output buffer. The Store module writes the final result from the output buffer back to external global storage.

Images