A neural network matrix multiplication accelerator and a deployment method thereof

Abstract

The application relates to the field of neural network inference calculation of artificial intelligence, and provides a neural network matrix multiplication accelerator and a deployment method thereof, aiming at improving the calculation efficiency in the field of artificial intelligence neural network inference calculation. The accelerator is designed based on a systolic array and comprises a systolic array calculation module, a control instruction chain module and a unified cache and memory read-write module. The application also provides a calculation method for improving the configuration value of a control register of a process element (PE) in the systolic array, a sorting configuration item calculation scheme of a reordering module and a scheduling scheme of the accelerator. The accelerator has universality, can perform matrix multiplication operation of any mask under the same hardware structure, effectively utilizes the high concurrency characteristics of an FPGA, reduces the calculation time and improves the energy efficiency ratio.

Description

Technical Field

[0001] This invention relates to the field of neural network inference computation in artificial intelligence, and in particular to a neural network matrix multiplication accelerator and its deployment method. Background Technology

[0002] With the development of Digital Signal Processing (DSP) technology, matrix multiplication has become a fundamental and crucial operation in this field. Especially in deep learning, matrix multiplication is the core of implementing various neural network algorithm models. While traditional CPUs and GPUs can handle these operations, their performance bottlenecks become increasingly apparent as model sizes continue to grow. Therefore, to improve computational efficiency, unstructured pruning techniques and corresponding hardware accelerator designs have emerged.

[0003] Unstructured pruning is a technique that prunes the weights of a deep learning model one by one. Compared to structured pruning, it can achieve higher compression ratios and speedups with less sacrifice in model performance. This pruning method directly modifies the weights, resulting in a nearly random distribution of 0 values ​​after pruning, with no discernible pattern. Although unstructured pruning can be more complex to deploy, it offers greater flexibility, allowing for customized optimizations for specific applications.

[0004] However, unstructured pruned matrices exhibit high randomness, requiring hardware platforms capable of efficiently handling operations on these matrices. Traditional CPUs and GPUs, whose architectures are not specifically designed for this type of matrix structure, cannot fully leverage the advantages of unstructured pruning. This underscores the necessity of designing dedicated hardware accelerators.

[0005] Customized hardware accelerators can be designed to meet specific algorithm requirements, maximizing performance and efficiency. By utilizing technologies such as systolic arrays, hardware accelerators can perform parallel processing efficiently, improving throughput. Furthermore, optimized hardware design can reduce power consumption during computation, making them suitable for applications with stringent energy efficiency requirements. More importantly, by improving systolic array technology, hardware accelerators can be specifically designed to support operations on truncated matrices, further enhancing computational efficiency.

[0006] The paper (Xu R, Ma S, Wang Y, et al. CMSA: Configurable multi-directional systolic array for convolutional neural networks [C] / / 2020 IEEE 38th International Conference on Computer Design (ICCD). IEEE, 2020: 494-497) proposes a hardware accelerator for systolic array computation of convolutional neural networks. Through configurable input and output direction processing units (Process Elements, PEs), this accelerator can achieve better performance in ordinary convolution and channel-wise convolution operations, but it does not have a specific design for quantization and pruning. In addition, patents CN116820743A and CN118797227A are both methods for optimizing the multiplication of sparse matrices, but they do not involve the operation scheduling of dense matrices with unstructured pruning. Summary of the Invention

[0007] For ease of description later, all symbols used in this article and their meanings are listed here:

[0008]

[0009] Continued table

[0010]

[0011] This invention aims to address the aforementioned problems by proposing a novel matrix multiplication hardware accelerator. This accelerator enables the deployment and optimization of matrix multiplication operations with arbitrary masks within the same hardware architecture. It also incorporates a unified caching system to reduce the accelerator's memory requirements. The invention employs the following technical solutions to address the aforementioned problems:

[0012] According to the present invention, a matrix multiplication accelerator based on a systolic array includes a systolic array computation module, a control instruction chain module, and a unified cache and memory read / write module; wherein,

[0013] The control instruction chain module is used to receive control operations from the CPU and coordinate the synchronization issues when various modules are operating in parallel.

[0014] The unified caching and memory read / write module is used to read from memory, temporarily store various configuration items and input matrices, and write output matrices to memory.

[0015] The pulsating array computation module consists of a configurable r-row c-column improved PE, an input synchronization module, an output synchronization module, and a reordering module. It receives PE configuration items and an input matrix from a unified cache module. The configuration items are loaded into the PE module and remain there. The input matrix flows between PE modules, undergoing multiplication with the weights in the configuration items. The output result flows within the PE module in a direction orthogonal to the input and is output to the output synchronization module.

[0016] The input synchronization module in the systolic array computing module is used to apply different delays to the input of the same clock cycle in order to match the timing requirements of the systolic array.

[0017] The output synchronization module in the systolic array computing module is used to accept the synchronization delay configuration item from the unified cache module, and delay each row of the systolic array output result by different periods to form an output matrix, which is then sent to the reordering module.

[0018] The reordering module in the pulsating array computing module is used to accept sorting configuration items from the unified cache module, reorder the rows of the output matrix in a hardware parallel bitone sorting manner, and output the sorting results to the unified cache.

[0019] The improved PE in the systolic array computation module adds a pipeline bypass control register on the input pipeline path to bypass the input data register, and a computation bypass control register to simultaneously control the adder section and the input and output registers. When the two bypass control registers are active, the output and input of the corresponding register are shorted and the enable terminal is invalidated, and the corresponding input terminal receives a logic 0.

[0020] As a method for calculating the configuration value of the PE control register in the pulsating array of the matrix multiplication accelerator described in this invention, the values ​​of the computation bypass register and the pipeline bypass register in the PE are obtained by performing row rearrangement and pipeline delay calculation on the unstructured pruned weight matrix, including the following steps:

[0021] Step 1: Based on the original weight matrix W org Calculate the original mask matrix M org ;

[0022] Step 2: Calculate the original mask matrix M org The sorting weight bin for each row (i) (1≤i≤r);

[0023] Step 3: Based on the sorting weight bin (i) Calculate the row swap index value k of the matrix i (1≤i≤r);

[0024] Step 4: Swap index values ​​k according to the matrix rows. i The original mask matrix M org and the original weight matrix W org The corresponding rows are reordered to obtain a new mask matrix M and a weight matrix W;

[0025] Step 5: Perform pipelined delay calculation on the mask matrix M and weight matrix W obtained in Step 4 to obtain the initial delay matrix D;

[0026] Step 6: Based on the physical characteristics of the PE in the actual chip, perform timing correction on the initial delay matrix D obtained in Step 5 to obtain the corrected delay matrix D. ′ ;

[0027] Step 7: Based on the corrected delay matrix D obtained in Step 6 ′ Calculate the bypass matrix B corresponding to the value of the bypass register in each PE. C and the pipeline bypass matrix B corresponding to the values ​​of the pipeline bypass registers in each PE. F .

[0028] As the calculation scheme for the sorting configuration item of the reordering module of the matrix multiplication accelerator described in this invention, the row exchange index value k obtained in step 3 of the calculation method based on the configuration value of the PE control register is used. i The algorithm pads the index values ​​to powers of 2 and sorts them. By recording whether or not the index values ​​are swapped during each comparison when sorting the padded index values, the sorting configuration options of the resorting module are obtained.

[0029] As a scheduling scheme for the matrix multiplication accelerator described in this invention, the input and weight matrices are divided into blocks according to the size of the systolic array hardware. For each sub-matrix of the weights, the values ​​of the two configuration registers in the PE and the sorting configuration items of the reordering module are pre-calculated. In actual calculation, each block is enumerated in turn. First, the weights and the pre-calculated configuration items are loaded into the corresponding hardware. Then, the input sub-matrix is ​​sent to the input end of the systolic array, and the output is received from the reordering module.

[0030] The main advantages of this invention include:

[0031] 1. The method of the present invention is universal and can perform dense matrix multiplication of arbitrary size by block calculation under the same hardware accelerator structure;

[0032] 2. This invention can effectively utilize the high concurrency characteristics of FPGA. By utilizing the high bit width of each bus inside the accelerator and the high parallelism of each module, the matrix multiplication calculation time can be significantly reduced.

[0033] 3. By preprocessing the weight matrix after unstructured pruning and bypassing some pipeline registers during computation, this invention can achieve a shorter total computation time during neural network computation and inference compared to the same hardware structure with the same resource usage and frequency without bypassing. Attached Figure Description

[0034] Figure 1 A diagram showing the internal structure of an improved PE in a hardware accelerator;

[0035] Figure 2 A block diagram of the overall structure of the hardware accelerator;

[0036] Figure 3 This is a block diagram of the pulsating array computing module in a hardware accelerator.

[0037] Figure 4 This is a block diagram of the organizational structure of a single row of PEs within the pulsating array computing module of a hardware accelerator. Detailed Implementation

[0038] Figure 1 The diagram shows the internal structure of the improved PE (Programmable Execution Unit) of the matrix multiplication accelerator described in this invention. It includes a multiplier, an adder, two computation registers, three configuration registers, and four multiplexers. When the computation bypass control register is enabled, the input multiplexer selects logic zero as its input, and the partial sum-out multiplexer selects the partial sum-in port of the PE as its input. When the computation bypass control register is disabled, the input multiplexer selects the input port of the PE as its input, and the partial sum-out multiplexer selects the output of the adder as its input. When the pipeline bypass control register is enabled, the input-out multiplexer selects the input port of the PE as its input; when the pipeline bypass control register is disabled, the input-out multiplexer selects the output of the input register as its input.

[0039] Figure 2 The diagram shows the overall architecture of the matrix multiplication accelerator described in this invention. The accelerator provides an Advanced eXtensible Interface (AXI) slave module, which connects to and is controlled by the CPU; it also provides an AXI master interface, which connects to memory and reads and writes data to it. The accelerator integrates a unified cache module and a systolic array module. After the CPU issues computation instructions to the accelerator, the instructions are first cached in a First-In-First-Out (FIFO) queue, and then flow sequentially to the subsequent control chain.

[0040] The method for calculating the configuration value of the PE control register in the improved pulsating array of the matrix multiplication accelerator described in this invention includes the following steps:

[0041] Step 1: Based on the original weight matrix W org The value is obtained using the formula:

[0042] M org(i,j) =E[W org(i,j) ≠0], 1≤i≤r, 1≤j≤c

[0043] Calculate the original mask matrix M org The original weight matrix W org Let the element be in the row of r and column of c, and the element in the i-th row and j-th column be W. org(i,j) (1≤i≤r, 1≤j≤c), E[x] equals 1 when x is true, otherwise it equals 0;

[0044] Step 2: Using the formula, on a per-row basis:

[0045]

[0046] Calculate the original mask matrix M org The sorting weight bin for each row (i) ;

[0047] Step 3: Based on the sorting weights bin obtained in Step 2 (i) The formula used is:

[0048]

[0049] Calculate the row swap index value k of the matrix i ;

[0050] Step 4: Convert the original mask matrix M org and the original weight matrix W org For the corresponding row, use the formula:

[0051]

[0052] The layers are reordered to obtain a new mask matrix M and a weight matrix W, where M... (i) Let be the i-th row of the mask matrix M (1≤i≤r);

[0053] Step 5: Apply the following formula to the mask matrix M and weight matrix W obtained in Step 4:

[0054]

[0055] Perform pipeline delay calculations to obtain the initial delay matrix D, where D (i,j) This represents the i-th row and j-th column of D;

[0056] Step 6: Perform timing correction on the initial delay matrix D obtained in Step 5. Specifically, based on the physical characteristics of the PEs in the actual chip, calculate the maximum number of continuous pipelined bypass PEs n that will not cause timing violations. v With the maximum continuous calculation of bypass PE number n h Then use the formula:

[0057]

[0058] After performing delay correction, the corrected delay matrix D is obtained. ′ ;

[0059] Step 7: Based on the corrected delay matrix D obtained in Step 6 ′ The formulas are as follows:

[0060] B C(i,j) =E[M (i,j) =0 and(j=1 or D′) (i,j) =D′ (i,j-1) ]

[0061] B F(i,j) :=E[i=r or D′ (i,j) =D′ (i+1,j) ]

[0062] Obtain the bypass matrix B C and the bypass matrix B F Among them, B C(i,j) =1 indicates that the PE calculation bypass control register in the i-th row and j-th column of the pulsating array is valid, B F(i,j) =1 indicates that the PE flow bypass control register in the i-th row and j-th column of the pulsating array is valid.

[0063] The internal structure of the reordering module of the matrix multiplication accelerator described in this invention adopts the structure of a Batcher merging network, in which the reference signal exchanged by each Batcher comparator is an external input signal configured in a register.

[0064] The calculation scheme for the sorting configuration item of the reordering module of the matrix multiplication accelerator described in this invention includes the following steps:

[0065] Step 1: Swap the matrix row index value k obtained when changing the value of the PE control register. i (1≤i≤c), use the formula:

[0066]

[0067] Add to the powers of 2, where B is the smallest power of 2 that is not less than c;

[0068] Step 2: Swap the index values ​​k′ of the rows in the expanded matrix. i Perform a bitonic sort (1≤i≤B), and record whether a swap is performed during each comparison;

[0069] Step 3: Save the data recorded in Step 2 as a configuration item and load it into the corresponding register of the reordering module when needed.

[0070] The scheduling scheme for the matrix multiplication accelerator described in this invention includes the following steps:

[0071] Step 1: Calculate the overall weight matrix W that actually participates in the calculation. A With the overall input matrix X A Divide the array into blocks according to the size of the pulsation array to obtain a series of sub-matrices W. A(i) ,X A(i) (1≤i≤n);

[0072] Step 2: Enumerate each weight submatrix W sequentially. A(i) (1≤i≤n), calculate the matrix multiplication corresponding to the submatrix and save the result to a unified cache;

[0073] Step 3: Send the matrix multiplication calculation results obtained in Step 2 out of the accelerator.

[0074] Step 2 in the scheduling scheme of the matrix multiplication accelerator described in this invention includes the following sub-steps:

[0075] Step 2a, for W A(i) It calculates the value of each configuration item and loads the weight matrix and its configuration items into the accelerator's unified cache;

[0076] Step 2b: Load the weight matrix and configuration items into the systolic array and reordering module;

[0077] Step 2c: When performing matrix multiplication, it is necessary to combine W... A(i) All input matrices X to be multiplied A(j) (1≤j≤n) are loaded into a unified cache in sequence, then sent to a systolic array, and the corresponding matrix multiplication is calculated.

[0078] Step 2d: Collect the corresponding multiplication result Y A(i,j) (1≤i,j≤n), save to a unified cache.

[0079] The complete instruction for a single operation of the matrix multiplication accelerator described in this invention includes:

[0080] 1. Read the enable flag, memory read address, and uniform cache offset of the control module;

[0081] 2. Load the enable flag of the control module, the offset of the unified cache for this load, the starting address of the systolic array configuration item, and the load length;

[0082] 3. Calculate the enable flag of the control module, the uniform buffer offset of the input matrix, the uniform buffer offset of the output matrix, and the data length;

[0083] 4. Write down the enable flag, uniform cache offset, memory write address and length of the control module.

[0084] The control logic and data flow scheme for a single operation of the matrix multiplication accelerator described in this invention include the following steps:

[0085] Step 1: The CPU writes the corresponding instruction from the module to the instruction FIFO via AXI;

[0086] Step 2: When the instruction FIFO has data and the read control module is idle, the instruction FIFO sends the first instruction to the read control module. Step 3: The read control module performs the data read operation based on the enable flag, memory read address, and unified cache offset in the instruction. When the enable flag is invalid, the read operation is completed immediately; otherwise, the AXI read address (AR) channel and read data (R) channel are used to read data of the corresponding position and length from memory and store it in the unified cache module.

[0087] Step 4: After the read control module operation is completed, when the loading control module is idle, send an instruction to the loading control module to load the control module, calculate the control module, and write out all data related to the control module, and restore the idle state after sending.

[0088] Step 5: The loading control module performs data loading operations based on the enabled flag of the loading control module, the unified cache offset for this loading, the starting address of the systolic array configuration item, and the loading length in the received instruction. When the enabled flag is invalid, the loading operation is completed immediately; otherwise, the configuration item specified by the instruction is sent to the systolic array from the unified cache offset for this loading.

[0089] Step 6: After the loading control module operation is completed, when the calculation control module is idle, send an instruction to the calculation control module to write out all data related to the control module, and restore the idle state after sending.

[0090] Step 7: The calculation control module performs calculations based on the enabled flag of the calculation control module, the unified buffer offset of the input matrix, the unified buffer offset of the output matrix, and the data length in the received instructions. When the enabled flag is invalid, the calculation operation is completed immediately; otherwise, starting from the unified buffer offset of the input matrix, all input data for this operation is sent to the systolic array, and starting from the unified buffer offset of the output matrix, all output data obtained from this calculation is received from the systolic array.

[0091] Step 8: After the calculation control module completes its operation, when the write control module is idle, send an instruction to the write control module to write all the data related to the control module, and restore the idle state after sending the instruction.

[0092] Step 9: The write control module performs a data write operation based on the enable flag, unified cache offset, memory write address, and length in the received instruction. When the enable flag is invalid, the write operation is completed immediately; otherwise, the data is written from the unified cache to memory starting from the unified cache offset and memory write address in the instruction through the AXI write address (AW) channel, write data (W) channel, and write response (B) channel.

[0093] It should be understood that the specific embodiments described herein are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A matrix multiplication hardware accelerator based on a pulsating array, characterized by the following modules: (1.1) Control instruction chain module, used to receive control operations from the CPU and coordinate the synchronization problem when various modules are operating in parallel; (1.2) Unified caching and memory read / write module, used to read from memory, temporarily store various configuration items and input matrices, and write output matrices to memory; (1.3) Pulsating array calculation module, used to accept PE configuration items and input matrix from unified cache module. The configuration items are stationary after being loaded into PE module. The input matrix flows between PE modules and is multiplied with the weights in the configuration items. The output result flows in PE module in the direction orthogonal to the input and is output to output synchronization module. Among them, (1.3) the pulsating array calculation module includes (1.4) the improved PE. Based on the traditional weighted pulsating array PE module design, a pipeline bypass control register for bypassing the input data register on the input pipeline path is added, as well as a calculation bypass control register that simultaneously controls the adder part and the input terminal, and the part and the output register. When the value of the two bypass control registers is 1, the output of the corresponding register is shorted to the input and the enable terminal is invalid, and the corresponding input terminal inputs logic 0. The improved PE configuration calculation method described in (1.4) is obtained through the following steps in (1.5): (1.5.1) Calculate the original mask matrix based on the original weight matrix; (1.5.2) Calculate the binary value corresponding to each row of the original mask matrix; (1.5.3) Calculate the row swap index value of the matrix based on the binary value corresponding to each row; (1.5.4) Reorder the corresponding rows of the original mask matrix and the original weight matrix according to the row swap index values ​​to obtain a new mask matrix and weight matrix; (1.5.5) Perform pipelined delay calculation on the mask matrix and weight matrix obtained in step (1.5.4) to obtain the initial delay matrix; (1.5.6) Based on the physical characteristics of the PE in the actual chip, the initial delay matrix obtained in step (1.5.5) is time-corrected to obtain the corrected delay matrix; (1.5.7) Based on the corrected delay matrix obtained in step (1.5.6), calculate the input bypass matrix corresponding to the value of the input bypass register in each PE and the output bypass matrix corresponding to the value of the output bypass register in each PE.

2. The matrix multiplication hardware accelerator based on a pulsating array according to claim 1, characterized in that, The pulsating array computing module also includes: (2.1) Input synchronization module, used to apply different delays to the input of the same beat to match the timing requirements of the pulsating array; (2.2) Output synchronization module, which is used to accept the synchronization delay configuration item from the unified cache module, and delay each row of the pulsating array output result by different periods to form an output matrix, which is then sent to the reordering module; (2.3) The reordering module is used to accept the sorting configuration items from the unified cache module, reorder the rows of the output matrix in a hardware parallel bitone sorting manner, and output the sorting results to the unified cache.

3. The matrix multiplication hardware accelerator based on a pulsating array according to claim 2, characterized in that, The calculation scheme for the sorting configuration item of the reordering module includes the following process: based on the obtained matrix row swap index value, pad it with an integer power of 2 and sort it. By recording whether or not the swap is performed during each comparison when sorting the padded index value, the sorting configuration item of the reordering module is obtained.

4. A matrix multiplication hardware accelerator based on a pulsating array according to claim 2, characterized in that... The scheduling scheme of the hardware accelerator includes the following process: After dividing the input and weight matrices into blocks according to the size of the systolic array hardware, for each sub-matrix of the weights, the values ​​of the two configuration registers in the PE and the sorting configuration items of the reordering module are pre-calculated; during actual calculation, each block is enumerated in turn, the weights and the pre-calculated configuration items are first loaded into the corresponding hardware, and then the input sub-matrix is ​​sent to the input end of the systolic array, and the output is received from the reordering module.

Images