Hybrid dimension acceleration operator design method based on input dimension separation

Abstract

The invention belongs to the field of FPGA hardware accelerator design, and discloses a mixed dimension acceleration operator design method based on input dimension separation. And taking a minimum convolution unit in the neural network as a basic convolution module, and performing function expansion on the basic convolution module to realize fusion of 2D convolution and 3D convolution. In addition, the sliding window thought of 3D convolution is optimized, dimension separation is carried out on the 3D feature map data to obtain multi-channel 2D feature map data, and parallel sliding window is carried out on the multi-channel 2D feature map data to obtain window data. And performing 2D convolution operation on the window data and the weight data after dimension separation to obtain a plurality of intermediate results, and screening and rearranging the intermediate results by an addition tree to obtain a final 3D convolution result. According to the design method, the resource occupation of the convolution module can be effectively reduced, so that the requirement of the hybrid convolution accelerator design on an FPGA development board is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of FPGA (Field-Programmable Gate Array) hardware accelerator design, and is particularly suitable for the design of a neural network accelerator having both 2D and 3D convolution structures. Background Art

[0002] In the field of aircraft engine fuel pump regulator fault diagnosis, convolutional neural network (CNN) plays an important role. It has powerful feature extraction capabilities, can mine deep features from the complex multi-source heterogeneous data of the regulator, reduce manual workload, and has a certain degree of robustness to data noise. It can accurately extract key fault features in noisy data; it can also process multiple types of data and integrate different data to assist in comprehensive diagnosis.

[0003] In the fault diagnosis of aircraft engine fuel pump regulators, 2D convolution is often used to analyze the surface image texture of components, the correlation between channels of two-dimensional images constructed by multiple sensors, etc., to determine the fault condition. At the same time, its computational complexity is relatively low, which makes it easy to deploy applications in scenarios with limited computing resources or high real-time requirements. However, 2D convolution cannot capture the time dimension information of the data, making it difficult to grasp the law of fault evolution over time, and its diagnostic capabilities are limited in complex situations involving multi-component coordination and fault spatiotemporal propagation.

[0004] The advantage of 3D convolution is its powerful ability to extract spatiotemporal features. It can process three-dimensional data structures and obtain spatial and temporal features at one time. It can accurately capture the mutual influence between the various components of the regulator at different times and the development and evolution of faults, and is also conducive to multi-source data fusion analysis. However, 3D convolution has high computational complexity and large amount of calculation, so it has strict requirements on hardware resources.

[0005] In order to improve the accuracy, comprehensiveness and reliability of aircraft engine fuel pump regulator fault diagnosis, the network structure using 2D and 3D hybrid convolution has become a new research direction, which has the following advantages: First, it can achieve more comprehensive feature extraction. After 3D convolution extracts the overall spatiotemporal features, 2D convolution further refines the local spatial details, making the extracted features richer, more accurate, and more in line with the actual fault situation. Secondly, it takes into account both computational efficiency and diagnostic accuracy, avoids the resource consumption and real-time problems caused by simple 3D convolution, and uses 2D convolution to improve computational efficiency. Furthermore, it can effectively improve the diagnostic capabilities of complex faults, understand the development mechanism of faults in the spatiotemporal dimension, and deal with complex fault scenarios.

[0006] In the computing field, common computing platforms include CPU (Central Processing Unit), GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), NPU (Neural Network Processing Unit), TPU (Tensor Processing Unit), DSP (Digital Signal Processing), and FPGA. FPGA has the characteristics of high flexibility and reconfigurability, low power consumption and high efficiency. Its powerful parallel processing capability is particularly suitable for image processing and edge computing accelerator design.

[0007] The design of FPGA accelerators involving 2D and 3D mixed convolutions usually adopts the strategy of 2D and 3D partition development, that is, a part of the resources on the FPGA development board is allocated to accelerate the 2D convolution layer, and the other part of the resources is allocated to accelerate the 3D convolution layer. This design method places high demands on the scale of the development board hardware resources and has a high implementation cost. In addition, 2D convolution generally uses the sliding window method to extract feature map data. The sliding window moves in the X and Y dimensions. After the X-dimensional sliding window extraction is completed, the coordinate value in the Y dimension is incremented and then the sliding window is performed again in the X dimension. The traditional 3D sliding window adopts the same idea as the 2D sliding window. After the sliding window in the X and Y directions is completed, the coordinate value in the Z dimension is incremented and the sliding window operation in the X and Y directions is repeated. The 3D sliding window under this idea will lead to a low reuse rate of feature map data. The feature map data needs to be read repeatedly, resulting in a waste of system bandwidth. Therefore, to design a relatively successful mixed-dimensional acceleration operator, it is necessary to take into account the parallelism, on-chip resource reuse rate, and feature map data reuse rate. This patent proposes a mixed-dimensional acceleration operator design based on input dimension separation, which has certain practical significance for the accelerated reasoning of mixed-dimensional neural networks on FPGA. Summary of the invention

[0008] In view of the above problems, the present invention proposes a mixed dimensional acceleration operator design scheme based on input dimension separation. Taking the minimum convolution unit in the neural network as the basic convolution module, the basic convolution module is functionally expanded to achieve the fusion of 2D convolution and 3D convolution. In addition, the present invention also optimizes the sliding window idea of ​​3D convolution, performs dimension separation on the 3D feature map data to obtain multi-channel 2D feature map data, and slides the multi-channel 2D feature map data in parallel to obtain window data. After performing 2D convolution operation on the window data and the weight data after dimension separation, several intermediate results are obtained, and the addition tree screens and rearranges the intermediate results to obtain the final 3D convolution result. This design method can effectively reduce the resource occupation of the convolution module, thereby reducing the requirements of such hybrid convolution accelerator design on the FPGA development board.

[0009] The technical solution of the present invention:

[0010] A mixed-dimensional acceleration operator design method based on input dimension separation includes a PS (Processing System) end and a PL (Programmable Logic) end. The PS end uses the GP interface and the HP interface to exchange data with the PL end; the PL end includes a convolution module, a weight loading module, a feature map loading module, a weight cache module, a feature map cache module, a feature map intermediate value temporary storage module, a control module, a DMA module, and PL_DDR (referring to the DDR of the hardware PL end, and the DDR of the PS end is referred to as PS_DDR below). The functions of each module are explained below:

[0011] 1. PS side: The PS side uses the GP interface to transmit command data and the HP interface to transmit feature map data and weight data. PS_DDR is used to store feature map data and weight data.

[0012] 2. Control module: responsible for judging and monitoring the stage of the convolution operation, and sending enable signals and instruction parameters to each module according to the progress of the operation, and configuring the operation mode of each module.

[0013] 3.PL_DDR: used to store feature map data, weight data and convolution operation results. When PL_DDR needs to update weight data and feature map data, the control module feeds back to the PS end through the GP interface, requesting the PS end to provide new weight data and feature map data.

[0014] 4. DMA module: responsible for moving data between PS_DDR and PL_DDR without the full participation of the PS side, reducing the performance occupancy of the PS side. After receiving the enable signal and instruction parameters from the control module, the DMA module starts to move the data of the instruction parameter length from the address specified by PS_DDR to PL_DDR, or starts to move the data of the instruction parameter length from the address specified by PL_DDR to PS_DDR. After the data is moved, the control module sends data and signals to the control module, and the control module then feeds back the execution status to the PS side, thus completing a complete data moving task.

[0015] 5. Weight loading module, weight cache module, weight storage module: The weight loading module is responsible for reading weight data from the address specified by PL_DDR into the weight cache module, and feeding back the status information to the control module. The weight cache module temporarily stores the weight data transmitted from the weight loading module, and updates the weight data to the weight storage module in turn according to the calculation progress and status.

[0016] 6. Feature map loading module, feature map cache module: The feature map loading module is responsible for reading feature map data from the address specified by PL_DDR into the feature map cache module, and at the same time feeding back the status information to the instruction interaction control module. The feature map cache module is responsible for inputting data into the convolution module.

[0017] 7. Filling module, sliding window module: The filling module is responsible for filling the feature map data input by the feature map cache module. The filled feature map data is passed to the sliding window module, which processes the multi-channel feature map data in parallel and generates multi-channel window data.

[0018] 8. Multiplication and addition module, addition tree module: The multiplication and addition module is responsible for using the on-chip multiplication resources to complete the multiplication and addition of the weight data and the corresponding window data and output the valid data to the addition tree module. After receiving the enable signal and instruction parameters from the control module, the addition tree module superimposes the valid data output by the multiplication and addition module according to the channel dimension according to the calculation mode to obtain the effective data of the addition tree.

[0019] 9. Feature map intermediate value temporary storage module: Due to hardware resource limitations, the hardware parallelism and the number of channels in the neural network cannot correspond one to one. To obtain a convolution result in a neural network, it is necessary to superimpose the results of multiple operations on the hardware side. The feature map intermediate value temporary storage module is responsible for temporarily storing the operation results output by the hardware side and superimposing multiple operation results. When the convolution result is obtained after superposition, the convolution result is output to PL_DDR.

[0020] A mixed-dimensional acceleration operator design based on input dimension separation includes the following steps:

[0021] Step S1: The PS side starts the DMA module to transfer the feature map data and weight data required by the convolutional neural network in PS_DDR to PL_DDR.

[0022] Step S2: Read out the weight data in PL_DDR. After the read weight data passes through the weight loading module and the weight cache module, the weight data is input from the weight cache module to the weight storage module in turn according to the calculation progress and status.

[0023] Step S3: Read out the feature map data of the neural network in PL_DDR in sequence, and input the read feature map data into the filling module after passing through the feature map loading module and the feature map cache module. The hardware input parallelism is pin. If the feature map data is 3D data with a size of w*h*d (width*height*depth), it is necessary to first split the 3D data into d 2D data of size w*h according to the d dimension, and then input them into the filling module in batches; if the data in the feature map intermediate value temporary storage module is 2D data, then directly input the 2D feature map data into the filling module in batches. Each batch inputs 2D feature map data of pin channels in parallel.

[0024] Step S4: The control module needs to determine whether the feature map data needs to be filled: if necessary, the filling module fills the data according to the instruction parameters, and outputs the processed data to the sliding window module; if not, the data is directly output to the sliding window module.

[0025] Step S5: The control module determines the operation mode to be performed by the sliding window module according to the operation progress, and sends an enable signal and a command parameter with sliding window size information to the sliding window module. The sliding window module processes the feature map data to obtain pin window data of size k*k, which are input into the multiplication and addition module.

[0026] Step S6: The control module sends the instruction parameters and enable signal with the calculation mode information (the calculation mode refers to 2D convolution or 3D convolution) to the multiplication and addition module according to the progress of the convolution operation. The multiplication and addition module arranges the weight data and window data according to the instruction parameters and performs multiplication and addition operations. The output parallelism of the hardware end is pout, and after the pin k*k window data undergo multiplication and addition operations, pin*pout convolution result intermediate values ​​are obtained.

[0027] Step S7: The control module sends the instruction parameters and enable signals with the calculation mode information to the addition tree module according to the progress of the convolution operation. The addition tree module rearranges the data of the intermediate values ​​of the pin*pout convolution results according to the instruction parameters and performs addition operations to obtain pout convolution results under the corresponding calculation mode.

[0028] Step S8: input the pout convolution results into the feature map intermediate value temporary storage module. At this time, the feature map intermediate value temporary storage module is only responsible for the temporary storage of data.

[0029] Step S9: Repeat steps S1 to S7. The control module determines the operation to be performed by the feature map intermediate value temporary storage module according to the progress of the convolution operation: read the value in the feature map intermediate value temporary storage module and add it with the new value, and write the accumulated result back to the feature map intermediate value temporary storage module; or output the result to PL_DDR after accumulation. The number of input channels of the convolutional neural network is input_channel, and the number of output channels of the convolutional neural network is output_channel. If the steps are repeated for the 2nd to input_channel / pin-1 rounds, and the final convolution result cannot be obtained after accumulation, the accumulated value is written back to the feature map intermediate value temporary storage module; if the steps are repeated to the input_channel / pin round, and the final value can be obtained after accumulation, the accumulated result is directly output to PL_DDR to obtain the final convolution result of pout channels.

[0030] Step S10: Repeat steps S1-S9 for a total of output_channel / pout times, and finally obtain the complete convolution result of output_channel channels. This result is used as the feature map data calculated by the next layer of network.

[0031] Step S11: Repeat steps S1 to S10 until all convolutional layers of the neural network are accelerated.

[0032] As described above, the present invention converts 3D data (including feature map data and weight data) into multi-channel 2D data based on input dimension separation, and sends the converted 2D data to the convolution module for calculation. The multiplication-addition module and the addition tree module in the convolution module first rearrange and then calculate the data in the input multiplication-addition module and the addition tree module according to the convolution operation mode (2D convolution or 3D convolution), and obtain the convolution results under different convolution modes. On the one hand, this method can integrate the resources of 2D convolution and 3D convolution to realize on-chip resource reuse, avoiding excessive resource occupancy caused by traditional 2D convolution and 3D convolution function partitioning; on the other hand, it optimizes the sliding window logic of 3D convolution, improves the data reuse rate of 3D feature map data and the efficiency of the sliding window module. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a design architecture diagram of a mixed-dimensional acceleration operator based on input dimension separation in the present invention. The thick arrows in the figure are data flows, and the thin arrows are instruction control signals. The dotted box is the convolution module, which is the core computing module of this accelerator.

[0034] Figure 2This is a diagram of the dimension separation rules and naming conventions for weight data in convolution calculations. Taking 2D convolution with 2 input parallelism and 2 output parallelism as an example, the weight data involved in obtaining output result 1 is called weight 1. Taking weight 4 as an example, dimension separation is performed, and it is consistent with feature Figure 1 The part of the convolution calculation is agreed to have a weight of 4_1, which is consistent with the feature Figure 2 The partial convention for convolution calculation is weight 4_2.

[0035] Figure 3 This is a diagram of the data arrangement of window data and weight data in the multiplication-addition module in the 3x3 convolution mode. There are 4 B modules in each channel, each B module consists of two A modules, and each A module is mapped to 3x3 multiplication units on the hardware side. The feature map data in the nth channel of the 2D feature map data is called feature map n, and the window data obtained after the feature map data passes through the sliding window module is named feature map n_window data. Figure 1 _ window data as an example, it needs to be calculated separately with the weight m_1 (m is 1 to 16), the feature Figure 1 The data arrangement of _window data and weight m_1 is shown in the figure. There are 8 channels in total, and the feature map p_window data and weight m_p are loaded in channel p (p is 1 to 8).

[0036] Figure 4 This is a diagram of the data arrangement of window data and weight data in the multiplication-addition module in the 5x5 convolution mode. There are 8 channels in total, and the feature map p_window data and weight m_p (m is 1 to 4) are loaded in channel p (p is 1 to 8). In each B module, a 5x5 weight is loaded by occupying 3 3x3 weight loading positions and filling the remaining positions with 0.

[0037] Figure 5 This is a schematic diagram of the structure of an addition tree cluster in the addition tree module. An addition tree cluster consists of two 8-1 addition trees and two 9-1 addition trees. An 8-1 addition tree consists of a 2-1 addition tree and two 3-1 addition trees; a 9-1 addition tree consists of three 3-1 addition trees.

[0038] Figure 6The diagram is a schematic diagram of the principle of converting 3D convolution into 2D convolution through dimension separation. The left side of the arrow is the dimension separation and naming convention of the feature map, window data, weight, and convolution result of the 3D convolution in the xyz coordinate system; the right side of the arrow is a schematic diagram of the sliding window of the 3D sliding window in the z direction in the yz coordinate system. The window data obtained after dimension separation, padding, and sliding window operation of the feature map are named as window data 1 to window data 12 in the z direction; the convolution result is named as convolution result 1 to convolution result 12 in the z direction after dimension separation; the weight is named as separated weight 1 to separated weight 3 in the z direction after dimension separation. In the yz plane, since the feature map is padded, 0 is filled before window data 1 and after window data 12 in the z direction. 2D convolution is performed between padding 0 and weight 1 after separation, 2D convolution is performed between window data 1 and weight 2 after separation, and 2D convolution is performed between window data 2 and weight 3 after separation. The above three convolution results are superimposed to obtain the value of convolution result 1 in the xyz coordinate system. Convolution results 2 to 12 can be obtained by analogy.

[0039] Figure 7 to Figure 9 Schematic diagram of the data arrangement of window data and weight data in the multiplication-addition module in the 3x3x3 convolution mode.

[0040] in Figure 7 Schematic diagram of the data arrangement of window data 1 and related weight data in 3D convolution; Fig. 9 Schematic diagram of data arrangement of window data 2 to window data 11 and related weight data in 3D convolution; Figure 8 Schematic diagram of the data arrangement of window data 12 and related weight data in 3D convolution. Figure 7 and Figure 8 There is only one channel, and each channel contains 2 B modules; Fig. 9 There are 10 channels in the , each channel contains 3 B modules. DETAILED DESCRIPTION

[0041] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0042] Take three convolution structures commonly used in the fault diagnosis network of aircraft engine fuel pump regulator (2D convolution 5x5 convolution, 3x3 convolution and 3D convolution 3x3x3 convolution) as an example. In order to further improve the real-time performance of the network and the parallelism of the accelerator, the weights are quantized to INT8 type in advance. In FPGA, a multiplication unit (DSP48E1, hereinafter referred to as dsp) can simultaneously calculate two INT8 multiplications of A*B and A*C in one cycle, where A, B, and C are all INT8 data. In this implementation, the padding of 5x5 convolution is 2, and the step size is 2; the padding of 3x3 convolution and 3x3x3 convolution is 1, and the step size is 1. Assume that the input feature map data size is 256*128*32 (width*height*number of channels), the result data a obtained after 3x3 convolution is 256*128*32 in size, and the result data b is obtained after 5x5 convolution, with a size of 128*64*32; multi-sensor data fusion on the aircraft engine fuel pump regulator obtains 2D data. Considering the time series characteristics of the data, the fused data of multiple consecutive moments are stacked to form a three-dimensional data body. A multi-channel three-dimensional data body will be generated during the neural network calculation process. Taking a three-dimensional data body of size 32*128*64*12 (number of channels*width*height*depth) as an example, the data body is convolved with 3x3x3 to obtain result data of size 32*128*64*12. In the hardware design, 3x3 convolution has 8 input parallelism and 16 output parallelism, that is, 8 input channels and 16 output channels, occupying 8x16x(3x3) / 2=576 dsp resources; 5x5 convolution has 8 input parallelism and 4 output parallelism, occupying 8x4x(3x(3x3)) / 2=432 dsp resources; in 3x3x3 convolution, 1 3D data is input and 4 3D data can be output after convolution, which is 1 input parallelism and 4 output parallelism (dimension separation of this process is equivalent to 12 input parallelism and 48 output parallelism under 2D convolution), using (3x3)x(10x3+2x2)x4 / 2=612 dsp resources. Since the convolution module adopts the design of integrating three convolution calculation modes, it only occupies 612 dsp units to realize the above three functions.

[0043] A.3x3 convolution step:

[0044] Step S1: The PS side starts the DMA module to transfer the feature map data and weight data required by the neural network in PS_DDR to PL_DDR.

[0045] Step S2: Read the weight data in PL_DDR. After the weight data passes through the weight loading module and the weight cache module, the weight data is loaded into the weight storage module for calculation. The multiplication and addition module reads the weight data in the weight storage module. In the 3x3 convolution mode, the input parallelism of the hardware end is 8, the output parallelism is 16, and the multiplication and addition module loads 16x8x3x3=1152 weight data.

[0046] Step S3: The feature map data in PL_DDR is read out in sequence. After passing through the feature map loading module and the feature map cache module, the read feature map data is input into the filling module in parallel with a parallelism of 8.

[0047] Step S4: the filling module needs to perform a filling operation with a filling value of 1 for the feature map data with an 8-channel size of 256*128, and obtains feature map data with an 8-channel size of 258*130 after filling. The filled feature map data is written into the sliding window module.

[0048] Step S5: The sliding window module performs a parallel sliding window operation with a step length of 1 on the feature map data with an input parallelism of 8, and inputs the window data into the multiplication and addition module along with the valid signal.

[0049] Step S6: The control module issues an instruction to the multiplication and addition module to perform a 3x3 convolution operation. The arrangement of the window data and weight data in the multiplication and addition module is as follows: Figure 3 The calculation results are output to the addition tree module.

[0050] Step S7: The control module sends an enable signal and instruction parameters to the addition tree module and configures it to a 3x3 convolution mode. Figure 5 For the addition tree module, when performing 3x3 convolution, the hardware end has 8 input parallelism and 16 output parallelism. It is necessary to add the 8 values ​​output by the multiplication and addition module to obtain an output result. Each output result requires an 8-1 addition tree or a 9-1 addition tree. In the present invention, 16 output results use 8 8-1 addition trees and 8 9-1 addition trees, for a total of 4 addition tree clusters. The result output by the multiplication and addition module is placed in the first level of the addition tree cluster, and the third level of the addition tree cluster outputs a valid result.

[0051] Step S8: the addition tree module inputs the result into the feature map intermediate value temporary storage module. At this time, the feature map intermediate value temporary storage module is only responsible for writing data.

[0052] Step S9: Repeat steps S1 to S7 for a total of four times. In the second to fourth repetitions, the feature map intermediate value temporary storage module needs to read out the data therein and add it to the new value output by the addition tree module. However, in the second and third repetitions, the result data after addition needs to be written back to the feature map intermediate value temporary storage module; the fourth time, the result after addition needs to be sent to the PL_DDR terminal. At this point, the output result data of 256*128*16 is obtained.

[0053] Step S10: Repeat steps S1 to S9 once to obtain the complete output data of 256*128*32, and a complete 3x3 convolution calculation is completed.

[0054] B.5x5 convolution step

[0055] Step S1: The PS side starts the DMA module to transfer the feature map data and weight data in PS_DDR to PL_DDR.

[0056] Step S2: Read the weight data in PL_DDR, and after passing through the weight loading module and the weight cache module, load the weight data into the weight storage module for calculation. The multiplication and addition module reads the weight data in the weight storage module. In the 5x5 convolution, the input parallelism of the hardware end is 8, and the output parallelism is 4. The multiplication and addition module needs to read 4x8x5x5=800 weight data.

[0057] Step S3: read out the feature map data in PL_DDR in sequence, and after passing through the feature map loading module and the feature map cache module, the read feature map data is input into the filling module in parallel with a parallelism of 8.

[0058] Step S4: The filling module needs to fill the feature map data with a filling value of 2, and write the feature map data into the sliding window module after filling. After filling, 8-channel feature map data with a size of 260*132 is obtained.

[0059] Step S5: The sliding window module performs a parallel sliding window operation with a step length of 2 on the input feature map data of the 8 channels, and inputs the valid window data into the multiplication and addition module along with the valid signal.

[0060] Step S6: The control module sends an enable signal and instruction parameters to enable the multiplication and addition module to perform a 5x5 convolution operation. The arrangement of the window data and weight data in the multiplication and addition module is as follows: Figure 4 The calculation results are output to the addition tree module.

[0061] Step S7: The control module sends an enable signal and instruction parameters to the addition tree module to configure its addition mode to be a 5x5 convolution mode. Figure 5For the addition tree module, when performing 5x5 convolution, the hardware side has 8 input parallelism and 4 output parallelism. It is necessary to add the 4 values ​​output by the multiplication and addition module to get an output result. Each output result requires an 8-1 addition tree. The result output by the multiplication and addition module is placed in the first level of the addition tree cluster, and the third level of the addition tree cluster outputs the valid result.

[0062] Step S8: the addition tree module inputs the result into the feature map intermediate value temporary storage module. At this time, the feature map intermediate value temporary storage module is only responsible for writing data.

[0063] Step S9: Repeat steps S1 to S7 for a total of four times. In the second to fourth repetitions, the feature map intermediate value temporary storage module needs to read out the data therein and add it to the corresponding new value output by the addition tree module. However, in the second and third repetitions, the result data after addition needs to be written back to the feature map intermediate value temporary storage module; the fourth time, the result data after addition needs to be sent to the PL_DDR terminal. At this point, the output result data of 128*64*4 is obtained.

[0064] Step S10: Repeat steps S1 to S9 for a total of eight times to obtain an output result of 128*64*32. At this point, the calculation of the 5x5 convolution is completed.

[0065] C. 3x3x3 convolution step

[0066] Step S1: The PS side starts the DMA module to transfer the feature map data and weight data in PS_DDR to PL_DDR.

[0067] Step S2: Read out the weight data in PL_DDR. After the read weight data passes through the weight loading module and the weight cache module, the weight data is loaded into the weight storage module for calculation. The multiplication and addition module reads the weight data in the weight storage module. In the 3x3x3 convolution, the hardware end inputs a 3D feature map data of size 128*64*12, and outputs 4 3D convolution result data of size 128*64*12. After dimension separation, the input parallelism of the hardware end is 12, and the output parallelism is 48. The multiplication and addition module needs to load 4x3x3x3=108 weight data.

[0068] Step S3: read out the feature map data of the neural network in PL_DDR in sequence, and after the read data passes through the feature map loading module and the feature map cache module, a 128*64*12 3D feature map data is converted into Figure 6 Dimension separation is performed along the z direction. After dimension separation, 12 channels of 2D feature map data with a size of 128*64 are obtained, and the dimension-separated data are input into the filling module in parallel.

[0069] Step S4: The filling module needs to perform a parallel filling operation for the 2D data of the 12 channels with a filling value of 1, and write the data to the sliding window module after the filling is completed. The size of the filled data is 130*66*12 (width*height*number of channels).

[0070] Step S5: The sliding window module performs a parallel sliding window operation with a step size of 1 on the feature map data of the 12 channels, and inputs the valid window data into the multiplication and addition module along with the valid signal.

[0071] Step S6: The control module sends an enable signal and instruction parameters to enable the multiplication and addition module to perform a 3x3x3 convolution operation. The arrangement of the window data and weight data in the multiplication and addition module is as follows: Figure 7 to Figure 9 The calculation results are output to the addition tree module.

[0072] Step S7: The control module sends an enable signal and instruction parameters to the addition tree module to configure the addition mode to be a 3x3x3 convolution mode. Figure 6 It can be seen that the convolution result x (x is 1 to 12) is obtained by superimposing the three results obtained by convolving the three window data and the three separated weights. However, the data before window data 1 and after window data 12 are 0, so the convolution results 1 and 12 are obtained by adding two non-zero results, and the convolution results 2 to 11 are obtained by adding three non-zero results. Therefore, to obtain the convolution results 1 to 12, two 2-1 addition trees and ten 3-1 addition trees are required, that is, one addition tree cluster. The hardware end outputs 4 groups of such results, so 4 addition tree clusters are required.

[0073] Step S8: the addition tree module inputs the result into the feature map intermediate value temporary storage module. At this time, the feature map intermediate value temporary storage module is only responsible for writing data.

[0074] Step S9: Repeat steps S1 to S7 for a total of 32 times. In the 2nd to 32nd repetitions, the feature map intermediate value temporary storage module needs to read out the data therein and add it to the corresponding new value output by the addition tree module. However, in the 2nd to 31st repetitions, the result data after addition needs to be written back to the feature map intermediate value temporary storage module; the 32nd repetition needs to send the result data after addition to the PL_DDR terminal. At this point, the output result data of 4*128*64*12 is obtained.

[0075] Step S10: Repeat steps S1 to S9 seven times, a total of eight times, to obtain the full result of 32*128*64*12. At this point, the calculation of 3x3x3 convolution is completed.

Claims

1. A mixed-dimensional acceleration operator design method based on input dimension separation, characterized in that: The system corresponding to the mixed-dimensional acceleration operator design method includes a PS end and a PL end; PS side: PS side uses GP interface to transmit command data, uses HP interface to transmit feature map data and weight data, PS_DDR is used to store feature map data and weight data; The PL end includes the convolution module for neural network calculation, the weight loading module, the feature map loading module, the weight cache module, the feature map loading module, the feature map cache module, the feature map intermediate value temporary storage module, the control module, the DMA module, and the PL_DDR; the details are as follows: 1) Control module: responsible for judging and monitoring the stage of the convolution operation, and sending enable signals and instruction parameters to each module according to the operation progress, and configuring the operation mode of each module; when the weight data and feature map data in PL_DDR need to be updated, the control module feeds back to the PS end through the GP interface, requesting the PS end to provide new weight data and feature map data; 2) DMA module: responsible for transferring data between PS_DDR and PL_DDR, without the need for the full participation of the PS end; after receiving the control command signal and parameters from the control module, the DMA module starts to transfer the data of the command length from the address specified by PS_DDR to PL_DDR, or starts to transfer the data of the command length from the address specified by PL_DDR to PS_DDR; after the data transfer is completed, the control module sends data and signals to the control module, and the control module then feeds back the execution status to the PS end, completing a complete data transfer task; 3) Weight loading module, weight cache module: After receiving the enable signal and instruction parameters from the control module, the weight loading module reads the weight data from the address specified by PL_DDR to the weight cache module, and feeds back the status information to the control module; the weight cache module temporarily stores the weight data transmitted from the weight loading module, and updates the weight data to the convolution module in turn as needed according to the calculation progress and status; 4) Feature map loading module, feature map cache module: After receiving the enable signal and instruction parameters from the control module, the feature map loading module reads the feature map data from the address specified by PL_DDR to the feature map cache module, and then feeds back the status information to the instruction interaction control module at the same time; the feature map cache module temporarily stores the feature map data transmitted from the feature map loading module, and the feature map cache module waits to receive the enable signal and instruction parameters from the control module before inputting the data into the convolution module; 5) The convolution module includes a weight storage module, a padding module, a sliding window module, a multiplication-addition module and an addition tree module; 5.1) Weight storage module: The weight storage module is responsible for receiving the weight data output by the weight cache module. During the calculation process of the convolution module, the weight data is stored in the weight storage module, which is convenient for the multiplication and addition module to read the weight data; 5.2) Filling module, sliding window module: The filling module is responsible for filling the feature map data input by the feature map cache module after receiving the enable signal and instruction parameters from the control module; the filled feature map data is transmitted to the sliding window module, and the sliding window module generates window data according to the enable signal and instruction parameters given by the control module; 5.3) Multiplication and addition module, addition tree module: After receiving the enable signal and parameter instruction of the control module, the multiplication and addition module reads the window data in the weight storage module and the sliding window module, and uses the multiplication resources to complete the multiplication and addition of the weight data and the corresponding window data. The processing object of the multiplication and addition module is a plurality of 2D window data and 2D weight data. The 2D window data and the 2D weight data are multiplied and added to obtain a plurality of convolution result intermediate values; after receiving the enable signal and parameter instruction of the control module, the addition tree module adds the plurality of convolution result intermediate values ​​obtained by the multiplication and addition module according to the channel dimension to obtain the convolution result of some channels; 6) Feature map intermediate value temporary storage module: The control module sends out an enable signal and instruction parameters according to the progress of the convolution operation. The feature map intermediate value temporary storage module temporarily stores some channel convolution results output by the addition tree module, and when the addition tree outputs a new partial channel convolution result, the partial channel convolution result in the feature map intermediate value temporary storage module is read out as needed, and accumulated with the new partial channel convolution result output by the addition tree; when the convolution results of all channels are accumulated, the data in the feature map intermediate value temporary storage module is the final convolution result, and the final convolution result is input into PL_DDR.

2. The mixed-dimensional acceleration operator design method based on input dimension separation according to claim 1 is characterized in that: The multiplication-addition module and addition tree module in the convolution module take the multiplication-addition module and addition tree module under the k1*k1 convolution as the basic units, and expand the functions of the k2*k2 convolution and k1*k1*k1 convolution according to the calculation characteristics of the k1*k1 convolution; wherein, k1>1, k1 is an integer; k2>k1, k2 is an integer; the multiplication-addition module takes the k1*k1 convolution multiplication-addition module as the basic unit, and the k2*k2 convolution requires (k2*k2) / (k1*k1) k1*k1 convolution multiplication-addition modules; the k1*k1*k1 convolution requires k1 k1*k convolution multiplication-addition modules; when calculating the k1*k1*k1 convolution, the three-dimensional data of height w*width h*channel number c is divided into c two-dimensional data of size w*h from the channel direction, and the c divided two-dimensional data are The data are simultaneously input into the sliding window module to obtain two-dimensional window data, and the three-dimensional weight matrix of size k1*k1*k1 is divided into k1 two-dimensional weight matrices of k1xk1 in the c direction. The two-dimensional window data and the two-dimensional weight matrix are input into the multiplication and addition module to obtain several two-dimensional convolution results, and the several two-dimensional convolution results need to be processed by the addition tree module to obtain the three-dimensional convolution result; the addition tree module uses the addition tree module of k1*k1 convolution as the basic unit, and the k2*k2 convolution and k1*k1*k1 convolution call the addition tree module of k1*k1 convolution to complete the addition calculation; in order to ensure that the convolution module can simultaneously calculate the three types of convolutions of k1*k1 convolution, k2*k2 convolution and k1*k1*k1 convolution, it is necessary to rearrange the data input into the multiplication and addition module and the addition tree module to adapt to the structure of k1*k1 convolution.

3. The mixed dimensional acceleration operator design method based on input dimension separation according to claim 2 is characterized in that: Step S1: The PS side starts the DMA module to transfer the feature map data and weight data required by the convolutional neural network in PS_DDR to PL_DDR; Step S2: read out the weight data in PL_DDR, and after the read weight data passes through the weight loading module and the weight cache module, the weight data is input from the weight cache module to the weight storage module in turn according to the calculation progress and status; Step S3: read out the feature map data of the convolutional neural network in PL_DDR in sequence, and input the read feature map data into the filling module after passing through the feature map loading module and the feature map cache module; the hardware input parallelism is pin; if the feature map data is 3D data, the size is width w*height h*depth d, it is necessary to first split the 3D data into d 2D data of size w*h according to d dimensions, and then input them into the filling module in batches; if the data in the feature map intermediate value temporary storage module is 2D data, then directly input the 2D feature map data into the filling module in batches; each batch inputs 2D feature map data of pin channels in parallel; Step S4: The control module needs to determine whether the feature map data needs to be filled: if necessary, the filling module fills the data according to the instruction parameters, and outputs the processed data to the sliding window module; if not, the data is directly output to the sliding window module; Step S5: the control module determines the operation mode to be performed by the sliding window module according to the operation progress, and sends an enable signal and an instruction parameter with sliding window size information to the sliding window module; the sliding window module processes the feature map data to obtain pin window data of size k*k, and these window data are input into the multiplication and addition module; Step S6: The control module sends the instruction parameter and the enable signal with the calculation mode information to the multiplication and addition module according to the progress of the convolution operation. The multiplication and addition module arranges the weight data and the window data according to the instruction parameters and performs the multiplication and addition operation. The output parallelism of the hardware end is pout, and pin k*k window data are multiplied and added to obtain pin*pout convolution result intermediate values. The calculation mode refers to 2D convolution or 3D convolution. Step S7: the control module sends the instruction parameter and the enable signal with the calculation mode information to the addition tree module according to the progress of the convolution operation. The addition tree module rearranges the data of the intermediate values ​​of the pin*pout convolution results according to the instruction parameters and performs addition operation to obtain the pout convolution results under the corresponding calculation mode. Step S8: input the pout convolution results into the feature map intermediate value temporary storage module. At this time, the feature map intermediate value temporary storage module is only responsible for temporary storage of data; Step S9: repeat steps S1 to S7, and the control module determines the operation to be performed by the feature map intermediate value temporary storage module according to the progress of the convolution operation: read the value in the feature map intermediate value temporary storage module and add it with the new value, and write the accumulated result back to the feature map intermediate value temporary storage module; or output the result to PL_DDR after accumulation; the number of input channels of the convolution neural network is input_channel, and the number of output channels of the convolution neural network is output_channel; if the steps are repeated for the 2nd to input_channel / pin-1th rounds, and the final convolution result cannot be obtained after accumulation, the accumulated value is written back to the feature map intermediate value temporary storage module; if the steps are repeated to the input_channel / pinth round, and the final value is obtained after accumulation, the accumulated result is directly output to PL_DDR to obtain the final convolution result of pout channels; Step S10: Repeat steps S1-S9 for a total of output_channel / pout times, and finally obtain the complete convolution result of output_channel channels. This result is used as the feature map data calculated by the next layer of network; Step S11: Repeat steps S1 to S10 until all convolutional layers of the neural network are accelerated.

Images