Neural network matrix multiplication accelerator and deployment method thereof

Abstract

The invention relates to the field of artificial intelligence neural network reasoning calculation, provides a neural network matrix multiplication accelerator and a deployment method thereof, and aims to improve the calculation efficiency in the field of artificial intelligence neural network reasoning calculation. The accelerator is designed based on a systolic array and comprises a systolic array calculation module, a control instruction chain module and a unified cache and memory read-write module. The invention further provides a calculation method of a control register configuration value of an improved processing unit PE in the systolic array, a sorting configuration item calculation scheme of a resorting module and a scheduling scheme of an accelerator. The accelerator has universality, matrix multiplication of any mask can be carried out under the same hardware structure, the high concurrency characteristic of the FPGA is effectively utilized, the calculation time is shortened, and the energy efficiency ratio is improved.

Description

Technical Field

[0001] The present invention relates to the field of neural network reasoning calculation of artificial intelligence, and in particular to a neural network matrix multiplication accelerator and a deployment method thereof. Background Art

[0002] With the development of digital signal processing (DSP) technology, matrix multiplication has become a basic and critical operation in this field. Especially in the field of deep learning, matrix multiplication is the core of implementing various neural network algorithm models. Although traditional CPUs and GPUs can handle these operations, their performance bottlenecks become increasingly prominent as the scale of the model continues to expand. Therefore, in order to improve computing efficiency, unstructured pruning technology and supporting hardware accelerator design came into being.

[0003] Unstructured pruning is a technique that prunes the weights in a deep learning model one by one. Compared with structured pruning, it can achieve higher compression ratios and acceleration effects with less sacrifice in model performance. This pruning method directly modifies the weights, making the distribution of 0 values ​​after pruning almost random and irregular. Although unstructured pruning may be more complex to deploy, it provides greater flexibility and allows customized optimization for specific applications.

[0004] However, the matrix after unstructured cropping is highly random, which requires the hardware platform to be able to efficiently process the matrix operations after cropping. When processing such operations, traditional CPUs and GPUs cannot fully utilize the advantages of unstructured cropping because their architectures are not designed for the structured matrix. This highlights the necessity of designing specialized hardware accelerators.

[0005] Customized hardware accelerators can be customized for specific algorithm requirements to achieve maximum performance and efficiency. By utilizing technologies such as systolic arrays, hardware accelerators can efficiently perform parallel processing and improve throughput. In addition, by optimizing the hardware design, power consumption during the operation can be reduced, making it suitable for application scenarios with strict requirements on energy efficiency. More importantly, by improving the systolic array technology, hardware accelerators can be specifically designed to support the operation of cropped matrices, thereby further improving computing efficiency.

[0006] The literature (Xu R, Ma S, Wang Y, et al. CMSA: Configurable multi-directional systolic array for convolutional neural networks [C] / / 2020 IEEE 38th International Conference on Computer Design (ICCD). IEEE, 2020: 494-497) proposed a systolic array operation hardware accelerator for convolutional neural networks. Through the configurable input and output direction of the operation unit (Process Element, PE), the accelerator can achieve better performance in ordinary convolution and channel-by-channel convolution operations, but there is no targeted design for quantization and pruning; in addition, patents CN116820743A and CN118797227A are both methods for optimizing the multiplication of sparse matrices, but do not involve the operation scheduling of unstructured pruned dense matrices. Summary of the invention

[0007] To facilitate the following description, all the symbols used in this article and their meanings are listed here: Continued

[0008] The present invention aims to solve the above existing problems and propose a new matrix multiplication hardware accelerator, which can perform matrix multiplication operation deployment and optimization of arbitrary masks under the same hardware structure, and add a unified cache system to reduce the memory demand of the accelerator hardware. The present invention adopts the following technical solutions to solve the above problems:

[0009] A matrix multiplication accelerator based on a systolic array according to the present invention includes a systolic array computing module, a control instruction chain module, and a unified cache and memory read-write module; wherein,

[0010] The control instruction chain module is used to accept the control operation of the CPU and coordinate the synchronization issues of parallel operations of various modules;

[0011] Unified cache and memory read / write module, used to read and temporarily store various configuration items and input matrices from memory, and write output matrices to memory;

[0012] The systolic array computing module consists of a configurable r-row and c-column improved PE, as well as an input synchronization module, an output synchronization module, and a reordering module. It is used to accept PE configuration items and input matrices from the unified cache module. The configuration items are parked after being loaded into the PE module. The input matrix flows between PE modules and is multiplied with the weights in the configuration items. The output results flow in the PE module in a direction orthogonal to the input and are output to the output synchronization module. Among them,

[0013] The input synchronization module in the systolic array computing module is used to delay the input of the same beat differently to match the timing requirements of the systolic array;

[0014] The output synchronization module in the systolic array computing module is used to accept the synchronization delay configuration item from the unified cache module, and delay each row of the systolic array output result by different periods to form an output matrix, and send it to the reordering module;

[0015] The reordering module in the systolic array computing module is used to accept the sorting configuration items from the unified cache module, reorder the rows of the output matrix in a hardware parallel bitune sorting manner, and output the sorting results to the unified cache;

[0016] The improved PE in the systolic array computing module, based on the traditional weight fixed systolic array PE module design, adds a pipeline bypass control register on the input pipeline path for bypassing the input data register, and a computing bypass control register that simultaneously controls the adder part and the input end, part and output register. When the two bypass control registers are valid, the output and input of the corresponding register are short-circuited and the enable end is invalid, and the corresponding input end inputs logic 0.

[0017] As a method for calculating the configuration value of the PE control register in the systolic array of the matrix multiplication accelerator of the present invention, by performing row rearrangement and pipeline delay calculation on the weight matrix after unstructured clipping, the values ​​of the calculation bypass register and the pipeline bypass register in the PE are obtained, including the following steps:

[0018] Step 1: According to the original weight matrix W org Calculate the original mask matrix M org ;

[0019] Step 2: Calculate the original mask matrix M org The sort weight bin corresponding to each row (i) (1≤i≤r);

[0020] Step 3: Based on sorting weight bin (i) Calculate the matrix row swap index value k i (1≤i≤r);

[0021] Step 4: Exchange index value k according to matrix row i The original mask matrix M org and the original weight matrix W org Reorder the corresponding rows to obtain a new mask matrix M and weight matrix W;

[0022] Step 5: Perform pipeline delay calculation on the mask matrix M and weight matrix W obtained in step 4 to obtain an initial delay matrix D;

[0023] Step 6: According to the physical characteristics of the PE in the actual chip, the initial delay matrix D obtained in step 5 is corrected to obtain the corrected delay matrix D. ′ ;

[0024] Step 7: The modified delay matrix D obtained in step 6 ′ , calculate the calculation bypass matrix B corresponding to the value of the calculation bypass register in each PE C And the pipeline bypass matrix B corresponding to the value of the pipeline bypass register in each PE F .

[0025] As a calculation scheme for the sorting configuration items of the reordering module of the matrix multiplication accelerator of the present invention, the row exchange index value k obtained in step 3 of the configuration value calculation method of the PE control register is i , padded and sorted by integer powers of 2, and the sorting configuration items of the reordering module are obtained by recording whether the padded index values ​​are exchanged each time when they are compared when sorting.

[0026] As a scheduling scheme for the matrix multiplication accelerator described in the present invention, after the input and weight matrices are divided into blocks according to the size of the systolic array hardware, for each block of the weight sub-matrix, the values ​​of two configuration registers in the PE and the reordering module sorting configuration items are pre-calculated; during actual calculation, each block is enumerated in turn, the weight and the pre-calculated configuration items are first loaded into the corresponding hardware, and then the input sub-matrix is ​​sent to the input end of the systolic array, and the output is received from the reordering module.

[0027] The advantages of the present invention mainly include:

[0028] 1. The method of the present invention is universal and can calculate dense matrix multiplication of any size by block calculation under the same hardware accelerator structure;

[0029] 2. The present invention can effectively utilize the high concurrency characteristics of FPGA, and can significantly reduce the matrix multiplication calculation time by utilizing the high bit width of each bus inside the accelerator and the high parallelism of each module;

[0030] 3. The present invention preprocesses the weight matrix after unstructured pruning and bypasses some pipeline registers during calculation, so that when performing neural network calculation inference, the total calculation time can be shortened compared with the same hardware structure with the same resource usage and the same frequency without bypassing. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Improved internal structure diagram of PE in hardware accelerator;

[0032] Figure 2 It is the overall structural block diagram of the hardware accelerator;

[0033] Figure 3 It is a structural block diagram of the systolic array computing module in the hardware accelerator;

[0034] Figure 4 This is a block diagram of the organizational structure of a single row of PEs in a systolic array computing module in a hardware accelerator. DETAILED DESCRIPTION

[0035] Figure 1 The figure shows the internal structure of the improved PE of the matrix multiplication accelerator of the present invention, including a multiplier, an adder, two calculation registers, three configuration registers and four multiplexers. When the calculation bypass control register is configured to be valid, the input multiplexer will select logic zero as input, and the partial sum outflow multiplexer will select the partial sum inflow port of the PE as input; when the calculation bypass control register is configured to be invalid, the input multiplexer will select the input port of the PE as input, and the partial sum outflow multiplexer will select the output of the adder as input. When the pipeline bypass control register is configured to be valid, the input outflow multiplexer will select the input port of the PE as input; when the pipeline bypass control register is configured to be invalid, the input outflow multiplexer will select the output of the input register as input.

[0036] Figure 2 The figure shows the overall architecture of the matrix multiplication accelerator described in the present invention. The accelerator provides an Advanced eXtensible Interface (AXI) slave interface module, which is connected to the CPU and controlled by the CPU; it provides an AXI master interface, which is connected to the memory and reads data from and writes data to the memory. The accelerator integrates a unified cache module and a systolic array module. After the CPU sends a calculation instruction to the accelerator, the instruction will first be cached in a First In First Out (FIFO) queue, and then flow to the subsequent control chain in sequence.

[0037] The method for calculating the configuration value of the PE control register in the improved systolic array of the matrix multiplication accelerator of the present invention comprises the following steps: Step 1: According to the original weight matrix W org The value of is, using the formula: M org(i,j) :=E[W org(i,j) ≠0],1≤i≤r,1≤j≤c Calculate the original mask matrix M org , where the original weight matrix W org is r rows and c columns, and the element in row i and column j is W org(i,j) (1≤i≤r,1≤j≤c), E[x] is equal to 1 when x is true, otherwise it is equal to 0; Step 2: For each line, use the formula: Calculate the original mask matrix M org The sort weight bin corresponding to each row (i) ; Step 3: sort weight bins obtained in step 2 (i) , using the formula: Calculate the matrix row swap index value k i ; Step 4: Substitute the original mask matrix M org and the original weight matrix W org The corresponding row of , using the formula: Rearrange and obtain a new mask matrix M and weight matrix W, where M (i) is the i-th row of the mask matrix M (1≤i≤r); Step 5: For the mask matrix M and weight matrix W obtained in step 4, use the formula: Perform pipeline delay calculation to obtain the initial delay matrix D, where D (i,j) represents the i-th row and j-th column of D; Step 6: Perform timing correction on the initial delay matrix D obtained in step 5. Specifically, according to the physical characteristics of the PE in the actual chip, calculate the maximum number of continuous pipeline bypass PEs n without timing violations. v The maximum number of continuous bypass PEs n h , and then use the formula: Perform delay correction to obtain the corrected delay matrix D ′ ; Step 7: The modified delay matrix D obtained in step 6 ′ , respectively using the formula: B C(i,j) :=E[M (i,j) =0 and(j=1 or D′ (i,j) =D′ (i,j-1) ] B F(i,j) :=E[i=r or D′ (i,j) =D′ (i+1,j) ] Get the calculation bypass matrix B C and pipeline bypass matrix B F , where B C(i,j) =1 means that the PE calculation bypass control register of the i-th row and j-th column in the systolic array is valid, B F(i,j) =1 represents that the PE pipe bypass control register of the i-th row and j-th column in the systolic array is valid.

[0038] As the internal structure of the reordering module of the matrix multiplication accelerator described in the present invention, a structure of a Belcher merging network is adopted, and the reference signal exchanged by each Belcher comparator is an external input signal configured by a register.

[0039] The calculation scheme of the sorting configuration items of the reordering module of the matrix multiplication accelerator according to the present invention comprises the following steps: Step 1: Swap the matrix row index value k obtained by changing the value of the PE control register i (1≤i≤c), using the formula: complemented to an integer power of 2, where B is the smallest integer power of 2 that is not less than c; Step 2: Exchange the index value k′ of the expanded matrix row i (1≤i≤B) performs bitwise sorting and records whether an exchange is performed during each comparison; Step 3: Save the data recorded in step 2 as a configuration item and load it into the corresponding register of the reordering module when in use.

[0040] The scheduling scheme of the matrix multiplication accelerator of the present invention includes the following steps: Step 1: The overall weight matrix W that actually participates in the operation A With the overall input matrix X A Divide the systolic array into blocks according to its size, and obtain a series of sub-matrices W after division. A(i) ,XA(i) (1≤i≤n); Step 2: Enumerate each weight sub-matrix W in turn A(i) (1≤i≤n), calculate the matrix multiplication corresponding to the submatrix and save the result to the unified cache; Step 3: Send the matrix multiplication calculation result obtained in step 2 out of the accelerator.

[0041] Step 2 in the scheduling scheme of the matrix multiplication accelerator of the present invention includes the following sub-steps: Step 2a: W A(i) It calculates the value of each configuration item and loads the weight matrix and its configuration items into the unified cache of the accelerator; Step 2b, loading the weight matrix and configuration items into the systolic array and reordering module; Step 2c: When doing matrix multiplication, you need to compare W A(i) Multiply all input matrices X A(j) (1≤j≤n) are loaded into the unified cache in sequence, and then sent to the systolic array to calculate the corresponding matrix multiplication; Step 2d: Collect the corresponding multiplication results Y A(i,j) (1≤i,j≤n), saved to the unified cache.

[0042] The complete instruction of a single operation of the matrix multiplication accelerator of the present invention includes: 1. Read the control module's enable flag, memory read address, and unified cache offset; 2. Load the enable flag of the control module, the unified cache offset of this load, the starting address of the systolic array configuration item, and the load length; 3. Calculate the enable flag of the control module, the unified cache offset of the input matrix, the unified cache offset of the output matrix, and the data length; 4. Write out the control module's enable flag, unified cache offset, memory write address, and length.

[0043] The control logic and data flow scheme for a single operation of the matrix multiplication accelerator of the present invention includes the following steps: Step 1, the CPU writes the corresponding instruction to the instruction FIFO from the module through the AXI; Step 2: When the instruction FIFO has data and the read control module is idle, the instruction FIFO sends the first instruction to the read control module; Step 3: The read control module performs data reading operations according to the read control module enable flag, memory read address, and unified cache offset in the instruction. When the enable flag is invalid, the read operation is completed immediately; otherwise, the AXI read address (AR) channel and read data (R) channel are used to read data of the corresponding position and length from the memory and store them in the unified cache module; Step 4: After the operation of reading the control module is completed, when the loading control module is idle, send instructions to the loading control module to load the control module, calculate the control module and write out all data related to the control module, and restore the idle state after the sending is completed; Step 5: The loading control module performs data loading operations according to the enable flag of the loading control module in the received instruction, the unified cache offset of this loading, the starting address of the systolic array configuration item, and the loading length. When the enable flag is invalid, the loading operation is completed immediately; otherwise, the configuration item specified by the instruction is sent to the systolic array from the unified cache offset of this loading; Step 6: After the loading control module operation is completed, when the computing control module is idle, the computing control module and all data related to the writing control module in the instruction are sent to the computing control module, and the idle state is restored after the sending is completed; Step 7, the calculation control module performs calculation operations according to the enable flag of the calculation control module in the received instruction, the unified cache offset of the input matrix, the unified cache offset of the output matrix, and the data length. When the enable flag is invalid, the calculation operation is completed immediately; otherwise, starting from the unified cache offset of the input matrix, all input data of this time are sent to the systolic array, and starting from the unified cache offset of the output matrix, all output data obtained from this calculation are received from the systolic array; Step 8: After the operation of the calculation control module is completed, when the write control module is idle, send an instruction to the write control module to write all data related to the control module, and restore the idle state after the sending is completed; Step 9: The write control module performs data write operation according to the enable flag of the write control module, the unified cache offset, the memory write address and length in the received instruction. When the enable flag is invalid, the write operation is completed immediately; otherwise, the data is written from the unified cache to the memory through the write address (AW) channel, write data (W) channel and write response (B) channel of AXI, starting from the unified cache offset and the memory write address in the instruction.

[0044] It should be understood that the specific embodiments described herein are only preferred embodiments of the present invention and are not intended to limit the present invention in any form. Any person skilled in the art can, without departing from the scope of the technical solution of the present invention, use the methods and technical contents disclosed above to make many possible changes and modifications to the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A matrix multiplication hardware accelerator based on a systolic array, characterized by including the following modules: (1.1) Control instruction chain module, used to accept CPU control operations and coordinate synchronization issues when each module performs parallel operations; (1.2) Unified cache and memory read / write module, used to read and temporarily store various configuration items and input matrices from the memory, and write output matrices to the memory; (1.3) The systolic array computing module is used to accept PE configuration items and input matrices from the unified cache module. The configuration items are parked after being loaded into the PE module. The input matrix flows between PE modules and is multiplied with the weights in the configuration items. The output results flow in the PE module in a direction orthogonal to the input and are output to the output synchronization module.

2. A matrix multiplication hardware accelerator based on a systolic array according to claim 1, characterized in that: 1.3) The following modules within the systolic array computing module: (2.1) Input synchronization module, used to delay the input of the same beat differently to match the timing requirements of the systolic array; (2.2) Output synchronization module, used to accept the synchronization delay configuration items from the unified cache module, and delay each row of the systolic array output result by different periods to form an output matrix, and send it to the reordering module; (2.3) a reordering module, used to accept the sorting configuration items from the unified cache module, reorder the rows of the output matrix in a hardware parallel bitune sorting manner, and output the sorting results to the unified cache; (2.4) Improved PE. Based on the traditional weight-fixed systolic array PE module design, a pipeline bypass control register for bypassing the input data register on the input pipeline path is added, as well as a calculation bypass control register that simultaneously controls the adder part and the input end, part and output register. When the value of the two bypass control registers is 1, the output and input of the corresponding register are short-circuited and the enable end is invalid, and the corresponding input end inputs logic 0.

3. The systolic array computing module structure according to claim 2, characterized in that (2.4) The improved PE configuration calculation method includes the following steps: (3.1) Calculate the original mask matrix based on the original weight matrix; (3.2) Calculate the binary value corresponding to each row of the original mask matrix; (3.3) Calculate the matrix row swap index value based on the binary value corresponding to each row; (3.4) Reorder the corresponding rows of the original mask matrix and the original weight matrix according to the matrix row exchange index value to obtain a new mask matrix and weight matrix; (3.5) Perform pipeline delay calculation on the mask matrix and weight matrix obtained in step (3.4) to obtain an initial delay matrix; (3.6) According to the physical characteristics of the PE in the actual chip, the initial delay matrix obtained in step (3.5) is time-corrected to obtain a corrected delay matrix; (3.7) Based on the modified delay matrix obtained in step (3.6), calculate the input bypass matrix corresponding to the value of the input bypass register in each PE and the output bypass matrix corresponding to the value of the output bypass register in each PE.

4. The systolic array computing module structure according to claim 2 and the improved PE configuration calculation method according to claim 3 (2.4), characterized in that (2.3) The calculation scheme of the sorting configuration items of the reordering module includes the following process: according to the obtained matrix row exchange index value, it is padded and sorted by an integer power of 2, and the sorting configuration items of the reordering module are obtained by recording whether the padded index value is exchanged each time when comparing when sorting.

5. A matrix multiplication hardware accelerator based on a systolic array according to claim 1, characterized in that The scheduling scheme of the hardware accelerator includes the following process: after the input and weight matrices are divided into blocks according to the size of the systolic array hardware, for each sub-matrix of the weight, the values ​​of the two configuration registers in the PE and the configuration items of the reordering module are pre-calculated; during the actual calculation, each block is enumerated in turn, the weights and the pre-calculated configuration items are first loaded into the corresponding hardware, and then the input sub-matrix is ​​sent to the input end of the systolic array, and the output is received from the reordering module.

Images