Fully connected layer configuration recommendation and code generation method based on systolic array

Abstract

The invention aims to provide a recommendation configuration and hardware code generation function based on a 1D systolic array. Mainly after the maximum number of multipliers and the size of a full connection layer of an embedded platform and the maximum burst transmission frequency and the maximum transmission bandwidth of an ip core are specified, a module of a system is used for giving an allocation method, the total amount of resources needing to be consumed and the total calculation period of the embedded platform. And on this basis, a verilog code implemented by hardware is given through an automatic code generation script. Therefore, the purpose of accelerating the calculation of the full connection layer on the embedded platform is achieved.

Description

Technical Field

[0001] The present invention relates to a systolic array architecture, and in particular to a systolic array configuration recommendation and code generation method for a fully connected layer. Background Art

[0002] Neural network acceleration inference based on systolic arrays has always received widespread attention, resulting in some mature methods and theories. With the development of embedded platforms, the carrier of neural networks is no longer limited to personal computers with powerful computing power. However, previous methods and theories are not well applicable to embedded platforms, making how to deploy systolic arrays on resource-limited platforms to accelerate the calculation of neural networks an independent issue. As one of the most commonly used neuron connection methods in neural networks, the fully connected layer is widely used in convolutional neural networks, recurrent neural networks and other deep neural networks. The calculation of the fully connected layer can be equivalent to the product between an input vector and a weight matrix. Therefore, the input vector and weight matrix to be calculated can be divided into several parts, and the calculations can be performed simultaneously to achieve the purpose of accelerating the calculation. At present, the common method for accelerating the fully connected layer is the 1D systolic array, but the 1D systolic array is only a method, and it does not take into account the implementation on a specific platform. In addition, when using 1D systolic arrays to accelerate fully connected layers of different sizes on different embedded platforms, their basic structure is the same. Rewriting the code every time the platform or workload is changed is a lot of repetitive and inefficient. Writing scripts for automatic code generation can solve this problem. There is some discussion about automatic code generation under traditional systolic array architectures, but it obviously does not apply to 1D systolic arrays, so this is also an open problem. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a configuration recommendation and hardware code generation function based on 1D systolic array. Mainly by specifying the maximum number of multipliers of the embedded platform, the size of the fully connected layer, the maximum number of burst transmissions and the maximum transmission bandwidth of the IP core, the use system gives its allocation method, the total amount of resources to be consumed and the total calculation cycle. Based on this, the Verilog code of the hardware implementation is given through the automatic code generation script. Thereby achieving the purpose of accelerating the fully connected layer reasoning on the embedded platform.

[0004] Technical solution: A fully connected layer configuration recommendation and code generation method based on systolic array, including the following contents:

[0005] Assume that there is an embedded platform with a maximum number of multipliers of l, and we want to accelerate an input feature F on the platform. i , a single fully connected layer with output feature Fo, and the maximum burst transmission number of the ip core supported by the embedded platform is M trans, the maximum transmission bandwidth is b t , and the bit width of the input data is b w , the interval between two transmissions is t interval The entire calculation is eventually divided into S parts, S1 and S2 are two candidate division schemes. After receiving the division, the FPGA part first transmits the data to the control module, which stores the input data and weight data in their respective buffers. Finally, the buffer transmits the data to the systolic array. PE is the functional unit that performs calculations in the systolic array. It performs a multiplication and addition calculation in each cycle. nun It is the number of PE units contained in a single systolic array. total Represents the total time spent in the entire acceleration process, t trans Represents the time the input data takes to be transmitted, t delay Represents the time from the completion of data transmission to the completion of the calculation of the last PE unit, t add Represents the time it takes to add the calculation results of the PE unit using the addition tree. times The number of times to use total select transfers for input data.

[0006] Step 1: Calculate the maximum transmission bandwidth b t and input data width b w The maximum number of divisible shares S1 under the restriction is described by the following formula:

[0007]

[0008] After rounding down, S1 will be used as a candidate for the maximum divisible number S, and step 2 will be executed.

[0009] Step 2: Calculate the maximum number of multipliers M m and the number of output features F o The maximum divisibility number S2 under the restriction is described by the following formula:

[0010]

[0011] S2 after rounding down will be used as another candidate for the maximum divisible number S. Go to step 3.

[0012] Step 3: Select the smaller number between the two candidates calculated in step 1 and step 2 as the final number of splits S, that is:

[0013] S = min(S1, S2)

[0014] At this point, the number of PEs in the entire systolic array can be further calculated as:

[0015] PE num =S*Fo

[0016] That is, each 1D systolic array contains F o PE units, the entire systolic array has a total of S such 1D systolic arrays. Go to step 4.

[0017] Step 4: Calculate the total time taken for the entire speedup process under a single fully connected layer workload.

[0018] t total =t trans +t delay +t add

[0019] The transmission time t trans Depends on the number of times the transmission is required times and the maximum number of burst transmissions M trans , and the other two parameters are described by the following formula

[0020] t delay =F o

[0021]

[0022] If trans times ≤M trans , no transmission interval, enter situation 1; if trans times >M trans , there is a transmission gap, entering situation 2.

[0023] Case 1: No transmission interval

[0024] At this point, the total transmission time is the number of transmissions, that is:

[0025]

[0026] Case 2: There is a transmission gap

[0027] At this time, the entire transmission time is the sum of the data transmission time and the total interval of data sending, that is:

[0028] t trans =trans times *M trrans +(trans times -1)*t interval

[0029] Step 5: After receiving the input vector, the control module in the FPGA part fills the input vector and the weight matrix, filling the number of rows of the input vector and the weight matrix to an integer multiple of the number of partitions. The original input is a length of F iIf the vector is a vector, then P 0 elements need to be filled at the end of the original vector, where:

[0030] P=SF i %S

[0031] The weight matrix needs to be filled with P row 0 elements at the end. After filling, the control module sends the data to the buffer of the systolic array, and the systolic array uses the data in the buffer for calculation. After the calculation is completed, the calculation result is sent to the control module, and the control module sends it to the PS via DMA.

[0032] Step 6: Set the input data width to b w The S calculated in step 3 is passed as a parameter to the code generator to generate Verilog code.

[0033] For Step 1, Step 2, and Step 3:

[0034] Let the input vector be A, The weight matrix is ​​denoted as B, The output result is denoted as C. The relationship between them is:

[0035] C=A*B

[0036] After the input vector is divided, the number of elements contained in each portion is P s ,but:

[0037]

[0038] Divide A into S parts, namely:

[0039] {A1, A2, ..., A s =A

[0040] in Similarly, divide B into S parts, namely:

[0041] {B1, B2, ..., B S =B

[0042] in Thus:

[0043] C=A1*B1+A2*B2+…+A S *B S

[0044] For step 4:

[0045] The total acceleration time of the calculation is an ideal case without considering the error. After the actual layout and routing, some delay errors are likely to occur. Since the entire design adopts a pipeline structure, no matter whether the single calculation is one fully connected layer or multiple fully connected layers, the overall time-consuming calculation is correct. Also because of this pipeline structure t delay Always equal to Fo.

[0046] For step 5:

[0047] Filling the input vector is relatively simple, but the weight matrix is ​​stored in RAM, so filling it is more difficult. However, because the elements that need to be filled are 0, the weight output can be kept as 0 when the RAM read address has reached the boundary and the input data is still valid.

[0048] Furthermore, the hardware code functions to be implemented in step 6 are specifically as follows:

[0049] S610: Calculate the bit width of the output result w , if O w <16, then:

[0050] O w =b w *2+PE num

[0051] If O w >16, then O w Force truncation to 16 bits. Too much bit width will significantly increase the hardware resources consumed by the entire calculation.

[0052] S620: Define the parameters, input ports and output ports of the main module. The parameters include input data bit width b w , PE unit number PE num And the output result bit width is O w The input port has clock signal, reset signal, valid signal, input data and weight data, and the output port has calculation result and calculation completion signal.

[0053] S630: Loop S times, and generate a 1D systolic array each time. The 1D systolic array is constructed with a modular concept, and the systolic array module mainly includes a PE module and a FIFO delay module. PE is a computing unit that performs a multiplication and addition calculation (MAC) once per clock cycle; the FIFO delay module is responsible for delaying the output of the PE unit. S640: Give a specific FIFO module definition. The FIFO unit is used to delay the output result, and different FIFO depths represent different delay effects. Due to the data flow characteristics of the systolic array, the output results between different columns are not calculated at the same time, but subsequent calculations require them to give results at the same time, so FIFO is used to delay them. Specifically, the result of the first column needs to be delayed by PE num -1 clock cycle, the result of the second column is delayed by PE num -2 clock cycles, and so on, until the PE num -1 column delay 1 clock cycle, PE num The column is not delayed.

[0054] S650: Send the delayed result to the addition tree for addition to calculate the final result. The top-level size of the addition tree is S. The result is sent to the PS end through the control module to complete the entire calculation process.

[0055] The present invention proposes a method for recommending configuration and generating code for a fully connected layer based on a systolic array, which is different from other existing fully connected layer acceleration methods based on a systolic array. The main features are as follows: first, the method provides a method for using a 1D systolic array based on platform configuration; second, the method proposes an automatic code writing script based on the configuration.

[0056] The present invention has the following significant advantages: through the configuration module, the 1D systolic array for accelerating the fully connected layer can be flexibly deployed in different embedded platforms and other resource-constrained scenarios, reducing the deployment difficulty. By summarizing and extracting the commonalities between different systolic array codes and using automatic code generation, the development cost, development difficulty and development cycle are further reduced, so that engineers with poor hardware foundation can also successfully deploy. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the process of the present invention;

[0058] Figure 2 Schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0059] The specific implementation of the present invention will be described below in conjunction with the accompanying drawings.

[0060] like Figure 1As shown, the present invention designs a fully connected layer configuration recommendation and code generation system based on a systolic array, comprising the following steps:

[0061] If you want to accelerate a single-layer fully connected neural network with 16 input features and 2 output features on an embedded platform with 16 multipliers. The maximum burst transmission number of the IP core supported by the embedded platform is 32, and the maximum transmission bandwidth is 1024 bits. The bit width of the transmitted data is 8 bits, and the interval between two transmissions is 50 cycles.

[0062] Step 1: Calculate the maximum number of divisible shares S1 under the maximum transmission bandwidth of 1024 bits and the input data width of 8 bits, where:

[0063]

[0064] After rounding down, S1 will be used as a candidate for the maximum divisible number S, and step 2 will be executed.

[0065] Step 2: Calculate the maximum divisible number S2 under the constraints of the maximum number of multipliers 16 and the number of output features 2, where:

[0066]

[0067] S2 after rounding down will be used as another candidate for the maximum divisible number S. Go to step 3.

[0068] Step 3: Select the smaller number between the two candidates calculated in step 1 and step 2 as the final number of splits S, that is:

[0069] S = min (128, 4) = 4

[0070] At this point, the number of PEs in the entire systolic array can be further calculated as:

[0071] PE num =4*2=8

[0072] That is, each 1D systolic array contains 2 PE units, and the entire systolic array has a total of 4 such 1D systolic arrays. Go to step 4.

[0073] Step 4: Calculate the total time t taken for the entire acceleration process under a single fully connected layer workload total :

[0074] t total =t trans +t delay +t add

[0075] The transmission time t transDepends on the number of times the transmission is required times and the maximum number of burst transmissions M trans , the other two parameters are:

[0076]

[0077] t dday =2

[0078]

[0079] Then trans times <M trans , no transmission interval, enter situation 1.

[0080] At this time, the entire transmission time is the sum of the data transmission time and the total interval of data transmission, that is,

[0081]

[0082] Step 5: After receiving the input vector, the control module in the FPGA part fills the input vector and the weight matrix, filling the number of rows of the input vector and the weight matrix to an integer multiple of the number of partitions. The original input is a vector of length 16, where:

[0083] P=SF i %S=0

[0084] No vector padding is required. The weight matrix also does not need padding. After padding is completed, the control module sends the data to the buffer of the systolic array, and the systolic array uses the data in the buffer to perform calculations. After the calculation is completed, the calculation result is sent to the control module, and the control module sends it to the PS via DMA.

[0085] Step 6: Set the input data width to b w =8bit and PE calculated in step 3 num =8 is passed as a parameter to the code generator to generate executable Verilog code.

[0086] For Step 1, Step 2, and Step 3:

[0087] Let the input vector be A, The weight matrix is ​​denoted as B, The output result is denoted as C. The relationship between them is:

[0088] C=A*B

[0089] After the input vector is divided, the number of elements contained in each portion is P s ,but:

[0090]

[0091] Divide A into 4 parts, namely:

[0092] {A1, A2, ..., A4} = A

[0093] in Similarly, divide B into 4 parts, namely:

[0094] {B1, B2, ..., B4} = B

[0095] in Thus:

[0096] C=A1*B1+A2*B2+…+A4*B4

[0097] For step 4:

[0098] The total acceleration time of the calculation is an ideal case without considering the error. After the actual layout and routing, some delay errors are likely to occur. Since the entire design adopts a pipeline structure, no matter whether the single calculation is one fully connected layer or multiple fully connected layers, the overall time-consuming calculation is correct. Also because of this pipeline structure t delay Always equal to 2.

[0099] For step 5:

[0100] Filling the input vector is relatively simple, but the weight matrix is ​​stored in RAM, so filling it is more difficult. However, because it needs to be filled with 0 elements, the weight output can be kept as 0 when the RANM read address has reached the boundary and the input data is still valid.

[0101] Furthermore, the hardware code functions to be implemented in step 6 are specifically as follows:

[0102] S610: Calculate the bit width of the output result

[0103] O w =16

[0104] S620: Define the parameters, input ports and output ports of the main module. The parameters include input data bit width b w =8, PE unit number PE num =8 and the output result bit width is O w=16. When the PS sends data through DMA, the M_AXIS_0tvalid signal is high. The control module uses this signal to determine the state of data transmission. When the transmission is in progress, the control module stores the valid data in the buffer. When the M_AXIS_0_tvalid signal changes from high to low, it means that a transmission is completed. The control module pulls the trans_done signal to a high level to notify the data trimming module. The function of the data trimming module is to split and fill the data. It will split the data stored in the buffer into 4 parts and transfer them to the array_buffer of 4 1D systolic arrays. After completing the work, the data trimming module will set the fix_done signal to a high level to notify the systolic array to perform calculations.

[0105] S630: Loop 4 times, generating a 1D systolic array each time. The 1D systolic array is constructed with a modular concept. The systolic array module mainly includes a PE module and a FIFO delay module. PE is a computing unit that performs a multiplication-addition calculation (MAC) once per clock cycle; the FIFO delay module is responsible for delaying the output of the PE unit. When the systolic array sees that the fix_done signal is high, it starts calculating.

[0106] S640: Give a specific FIFO module definition. The FIFO unit is used to delay the output result, and different FIFO depths represent different delay effects. Due to the data flow characteristics of the systolic array, the output results between different columns are not calculated at the same time, but the subsequent calculations require them to give results at the same time, so FIFO is used to delay them. Specifically, the result of the first column is delayed by 1 clock cycle, and the result of the second column does not need to be delayed. After the delay, the systolic array pulls the array_done signal high to notify the subsequent addition tree module to perform calculations.

[0107] S650: Send the four vectors of size 2 after the delay to the addition tree module for addition and calculate the final result. After the addition tree module completes the calculation, it sets the add_done signal to a high level. After checking that the add_done signal is high, the data transmission module puts the two 16-bit results into the FIFO and transmits them to the DMA module in the form of a data stream. The DMA module then sends the results to the PS end to complete the entire calculation process.

[0108] The configuration recommendation and code generation script of this example are developed based on python3.9. The final generated Verilog code is verified in Vivado2021.1 and tested on the Xilinx xc7z020 development board. The final resource utilization information is listed in Table 1.

[0109] Table 1 Resource utilization

[0110] LUT FF BRAM URAM DSP 18900 31142 42.5 0 0

Claims

1. A fully connected layer configuration recommendation and code generation method based on a systolic array, characterized in that: The steps include: Step 1: Calculate the maximum transmission bandwidth b t and input data width b w The maximum number of divisible shares S1 under the restriction is described by the following formula: After rounding down, S1 will be used as a candidate for the maximum divisible number S; Step 2: Calculate the maximum number of multipliers M m and the number of output features F o The maximum divisibility number S2 under the restriction is described by the following formula: S2 after rounding down will be used as another candidate for the maximum divisible number S; Step 3: Select the smaller number between the two candidates calculated in step 1 and step 2 as the final number of splits, that is S = min(S1, S2) At this point, the number of PEs in the entire systolic array can be further calculated as: INSTEAD num =S*F o That is, each 1D systolic array contains F o PE units, the entire systolic array has a total of S such 1D systolic arrays; Step 4: Calculate the total time t taken for the entire acceleration process under a single fully connected layer workload total : t total =t trans +t delay +t add The transmission time t trans Depends on the number of times the transmission is required times and the maximum number of burst transmissions M trans , t delay =F o If trans times ≤M trans , no transmission interval, enter situation 1; if trans times >M trans , there is a transmission gap, entering situation 2; Case 1: No transmission interval At this time, the total transmission time is the number of transmissions, that is, Case 2: There is a transmission gap At this time, the total transmission time is the sum of the data transmission time and the total interval of data transmission. t trans =trans times *M trans +(trans times -1)*t interval Step 5: The control module of the hardware code fills the input vector and weight matrix with the number of rows of the input vector and weight matrix to be an integer multiple of the number of splits to avoid data ambiguity during acceleration. The original input is a length of F i If the vector is a vector, then P 0 elements need to be filled at the end of the original vector. P=S-F i %S The weight matrix needs to be filled with P rows of 0 elements at the end; Step 6: Set the input data width to b w and the PE calculated in step 3 num Passed as a parameter to the code generator to generate executable Verilog code.

2. A method for fully connected layer configuration recommendation and code generation based on a systolic array according to claim 1, characterized in that Steps 1, 2, and 3 specifically include: Let the input vector be A, The weight matrix is ​​denoted as B, The output result is denoted as C. The relationship between them is: C=A*B After the input vector is divided, the number of elements contained in each portion is P s ,but: Divide A into S parts, that is, <h2 style=";text-align:left;direction:ltr">{A1, A2,…, A<h2 style=";text-align:left;direction:ltr"> S <h2 style=";text-align:left;direction:ltr">}=A in Similarly, divide B into S parts, namely: {B1,B2,…,B S }=B in Thus: <h2 style=";text-align:left;direction:ltr">C = A1*B1+A2*B2+…+A<h2 style=";text-align:left;direction:ltr"> S <h2 style=";text-align:left;direction:ltr"> *B<h2 style=";text-align:left;direction:ltr"> S <h2 style=";text-align:left;direction:ltr"> 。 3. A method for fully connected layer configuration recommendation and code generation based on systolic array according to claim 1, characterized in that Step 4 specifically includes: The total acceleration time of the calculation is an ideal case without considering the error. After the actual layout and routing, some delay errors are likely to occur. Since the entire design adopts a pipeline structure, no matter whether the single calculation is a fully connected layer or multiple fully connected layers, the overall time-consuming calculation is correct. Also because of this pipeline structure, delay Always equal to F o .

4. A method for fully connected layer configuration recommendation and code generation based on systolic array according to claim 1, characterized in that Step 5 specifically includes: Filling the input vector is relatively simple, but the weight matrix is ​​stored in RAM, so filling it is more difficult. However, because the elements that need to be filled are 0, the weight output can be kept as 0 when the RAM read address has reached the boundary and the input data is still valid.

5. A method for fully connected layer configuration recommendation and code generation based on systolic array according to claim 1, characterized in that Step 6 specifically includes: S610: Calculate the bit width of the output result w , if O w <16, then: A w =b w *2+PE num If O w >16, then O w Force truncation to 16 bits. Too much bit width will significantly increase the hardware resources consumed by the entire calculation; S620: Define the parameters, input ports and output ports of the main module. The parameters include input data bit width b w , PE unit number PE num And the output result bit width is O w , the input port has clock signal, reset signal, valid signal, input data and weight data, and the output port has calculation result and calculation completion signal; S630: loop S times, generating a 1D systolic array each time. The 1D systolic array is constructed with a modular concept, and the systolic array module mainly includes a PE module and a FIFO delay module. PE is a computing unit that performs a multiplication and addition calculation (MAC) once per clock cycle; the FIFO delay module is responsible for delaying the output of the PE unit; S640: Give a specific FIFO module definition. The FIFO unit is used to delay the output result. Different FIFO depths indicate different delay effects. Due to the data flow characteristics of the systolic array, the output results of different columns are not calculated at the same time, but the subsequent calculations require them to give results at the same time, so FIFO is used to delay them. Specifically, the result of the first column needs to delay PE num -1 clock cycle, the result of the second column is delayed by PE num -2 clock cycles, and so on, until the PE num -1 column delay 1 clock cycle, PE num The column is not delayed; S650: Send the delayed result to the addition tree for addition, and calculate the final result, wherein the top-level size of the addition tree is S, and send the result to the PS end through the control module to complete the entire calculation process.

Images