A DCNN accelerator based on hybrid multi-line data flow strategy

Abstract

The present invention discloses a DCNN accelerator based on a hybrid multi-row data flow strategy, which is composed of a plurality of convolution processing modules stacked together. The convolution processing module comprises a plurality of parallel computing unit arrays, a computing buffer and a data buffer. The data transmission of adjacent convolution processing modules is in units of rows, and the row data is stored in the data buffer. The data of the data buffer is read in sequence and sent to the computing buffer after rearrangement operation for operation of the computing unit array. Each computing unit array is responsible for calculating the output feature map of a single row, and all computing unit arrays share the same weight data, and all weight data are stored in an off-chip DRAM. By adjusting the parallelism of the computing unit array of the convolution processing module, the off-chip bandwidth usage can be adjusted, thereby solving the problem that the existing layer-by-layer pipeline accelerator cannot achieve off-chip bandwidth optimization.

Description

Technical Field

[0001] The present invention relates to the field of electronic information and deep learning technology, and more specifically, to a DCNN accelerator based on a hybrid multi-row data flow strategy. Background Art

[0002] In the wave of artificial intelligence that has been developing in recent years, deep convolutional neural networks (DCNNs) have shown superior performance compared to traditional algorithms in the fields of object detection, semantic segmentation, face recognition, speech recognition, and medical auxiliary diagnosis. As a result, DCNNs have received widespread attention and research.

[0003] Since the intra-layer and inter-layer parallelism of the DCNN model can be fully utilized, the layer-by-layer pipeline system architecture is widely used in FPGA-based DCNN accelerators. In the layer-by-layer pipeline system architecture, the computational paradigm (row-by-row, layer-by-layer) of the convolutional layer computing task determines the number of times the weight data is read from the off-chip DRAM, which in turn determines the off-chip bandwidth of the accelerator. However, the existing layer-by-layer pipeline architectures all use a fixed computational paradigm, which makes the throughput performance of the accelerator subject to the off-chip bandwidth, and it is difficult to reduce the usage of off-chip bandwidth through on-chip storage.

[0004] The prior art discloses a convolution computing device based on hybrid parallelism, the device comprising: an input module, configured to obtain input convolution data and corresponding parameters, determine the convolution shape according to the input convolution data, and extract the feature map size, convolution kernel size and number of channels of the input convolution data; a simulation module, configured to obtain the parallelism corresponding to the input convolution data according to the data features extracted by the input module; the data features include the convolution shape and parameters; an on-chip processor, comprising a plurality of parallel processing modules; a grouping control module, connected to each processing module respectively, configured to divide all processing modules on the on-chip processor into G groups according to the parallelism, G equals the parallelism, and the number of processing modules in each group is equal; a mapping module, connected to each processing module respectively, configured to control the data and parameters input into each processing module according to the parallelism, the input convolution data and the corresponding parameters; wherein the processing modules in the same group input the same parameters and different data; the processing modules in different groups input different parameters; the processing modules are used to complete the convolution acceleration behavior and output the results according to the input data and parameters. This solution also makes it difficult to reduce the usage of off-chip bandwidth through on-chip storage. Summary of the invention

[0005] The present invention provides a DCNN accelerator based on a hybrid multi-row data flow strategy, which solves the problem that the existing layer-by-layer pipeline DCNN accelerator cannot achieve off-chip bandwidth optimization.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] A DCNN accelerator based on a hybrid multi-row data flow strategy includes a convolution processing unit and a fully connected processing unit, wherein:

[0008] The convolution processing unit is responsible for processing the convolution calculation part in the DCNN model. The convolution processing unit includes multiple convolution processing modules, bypass convolution processing modules and branch processing modules. The multiple convolution processing modules are connected in sequence. The number of convolution processing modules is equal to the number of convolution layers L of the DCNN model. The number of input row data of each convolution processing module is r. i , the number of output rows is r i+1 , and the number of input row data of the convolution processing module is the number of output row data of the previous convolution processing unit; the number of input row data of the bypass convolution processing module is the number of output row data of the first convolution processing module, the output of the bypass convolution processing module is the input of the branch processing module, the output of the branch processing module is the input of the last convolution processing module, and the branch processing module processes the branch part in the deep convolutional neural network;

[0009] The output row data of the last convolution processing module is output to the fully connected processing unit, and the fully connected processing unit is used to process the fully connected layer part in the deep convolutional neural network.

[0010] Preferably, a pooling processing module is connected between each convolution processing module, and the pooling processing module processes the pooling layer part in the deep convolutional neural network.

[0011] Preferably, the external data source inputs a row of input feature map data to the first convolution processing module every ΔT1 clock cycle, and every r i+1 S i ΔT i clock cycles, the i-th convolution processing module completes r i+1 The calculation of the row output feature map data, where

[0012]

[0013] Where Ps j is the stride of the pooling module after the jth convolutional processing module, S j is the stride of the input feature map corresponding to the jth convolution processing module, and the size of the convolution kernel corresponding to the input feature map is N i ×C i ×K i ×K i , padding is pad i .

[0014] Preferably, each of the convolution processing modules comprises an input data buffer, a plurality of parallel computing buffers, a plurality of parallel computing unit arrays, and an output data buffer, wherein:

[0015] The input data buffer reads and saves the data in the output data buffer of the previous convolution processing module, the multiple parallel computing buffers read the data in the input data buffer, the input of the multiple parallel computing unit arrays is the data in the computing buffer, and the output of the multiple parallel computing unit arrays is saved to the output data buffer.

[0016] The data transmission of adjacent convolution processing modules is in units of rows. The row data is stored in the data buffer. The data in the data buffer is read in sequence and sent to the calculation buffer after rearrangement for calculation of the calculation unit array. Each calculation unit array is responsible for calculating the output feature map of a single row. All calculation unit arrays share the same weight data, and all weight data are stored in the off-chip DRAM.

[0017] Preferably, the plurality of parallel computing unit arrays are composed of W h，i ×I w，i computing units, each of which is a W w，i The input multiplication-accumulation tree, the intermediate calculation data is cached in the dual port RAM, and the final calculation result is cached in the RAM, where the RAM serves as the data source of the input data buffer of the next convolution processing module.

[0018] Preferably, each computing unit performs small-size matrix multiplication W in sequence. rb ×I rpb , thus finally achieving large-size matrix multiplication W r ×I rp , the final calculation result is a single row of output feature map data.

[0019] Preferably, the computing paradigm of each computing unit is based on Toeplitz matrix multiplication, and the input feature map data is converted into a Toeplitz matrix. The input feature map data processed by each PE array is located in the column matrix of the Toeplitz matrix. In the mixed multi-row data flow strategy, all parallel computing unit arrays share the same weight parameters to separately process the Ifmap data from different column matrices in the Toeplitz matrix, thereby optimizing bandwidth usage.

[0020] Preferably, the computing resource usage #PE Mult of each computing unit is:

[0021] #PE Mult=W h，i ×Ww，i ×I w，i

[0022] Among them, W h，i , W w，i , I w，i Needs to be satisfied

[0023]

[0024] Where Hout i is the width and height of the output feature map corresponding to the i-th convolution processing module, and ΔT1 is the clock cycle interval for the external data source to input rows of data to the accelerator, satisfying:

[0025] TRP obj Expectations for designing for an accelerator

[0026] Preferably, the data stored in each input data buffer has the same row position and column position in the input feature map, and is arranged in the input data buffer according to the channel size. The number of data buffers in the i-th convolution processing module is #RowDataBuffer×Hin i :

[0027]

[0028]

[0029] DataIn0′=K i +S i (r i+1 -1)+GCD(r i+1 S i , r i )(r i ′-1)

[0030] Among them, GCD(r i+1 S i , r i ) is r i and r i+1 S i The greatest common divisor of i ′ and (r i+1 S i )′ are two positive integers that are mutually prime, padding is the step size corresponding to the output feature map, and padding is pad i .

[0031] Preferably, when the pooling processing module responsible for processing the pooling operation is used to process the output data of the convolution processing module, the number of its input row data is equal to the number of output row data of the convolution processing module, and the number of data buffers included satisfies:

[0032] Hout i ×#RowPoolingBuffer

[0033] in,

[0034] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0035] The present invention introduces a hybrid multi-row computing paradigm (data flow strategy) into the layer-by-layer pipeline system architecture. Through flexible data flow strategy configuration, efficient on-chip storage and off-chip bandwidth balance can be achieved, thereby improving the flexibility of the layer-by-layer pipeline system architecture design and its theoretical throughput upper limit. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the overall framework of the accelerator of the present invention.

[0037] Figure 2 Schematic diagram of mixed multi-row data flow strategy.

[0038] Figure 3 Schematic diagram of the convolution processing module.

[0039] Figure 4 Schematic diagram of the hardware structure of the computing unit array.

[0040] Figure 5 Schematic diagram of the data storage order of the input data buffer.

[0041] Figure 6 Schematic diagram of the convolution calculation paradigm based on the Toeplitz matrix.

[0042] Figure 7 Schematic diagram of the order in which weight data and Toeplitz matrix data are processed for the computational cell array. DETAILED DESCRIPTION

[0043] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0044] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0045] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0046] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0047] Example 1

[0048] This embodiment provides a DCNN accelerator for a hybrid multi-row data flow strategy, such as Figure 1 As shown, it includes a convolution processing unit and a fully connected processing unit, where:

[0049] The convolution processing unit is responsible for processing the convolution calculation part in the DCNN model. The convolution processing unit includes multiple convolution processing modules, bypass convolution processing modules and branch processing modules. The multiple convolution processing modules are connected in sequence. The number of convolution processing modules is equal to the number of convolution layers L of the DCNN model. The number of input row data of each convolution processing module is r. i , the number of output rows is r i+1 , and the number of input row data of the convolution processing module is the number of output row data of the previous convolution processing unit; the number of input row data of the bypass convolution processing module is the number of output row data of the first convolution processing module, the output of the bypass convolution processing module is the input of the branch processing module, the output of the branch processing module is the input of the last convolution processing module, and the branch processing module processes the branch part in the deep convolutional neural network;

[0050] The output row data of the last convolution processing module is output to the fully connected processing unit, and the fully connected processing unit is used to process the fully connected layer part in the deep convolutional neural network.

[0051] Figure 1 In the figure, BottleNeck is a branch processing module used in networks such as RESNET, FPM (Fully Connected Process Module) is a fully connected processing unit used to process the fully connected layer in the network; CPM (Convolution Process Module) is a convolution processing unit used to process the convolution calculation part in the network; External Memory is off-chip storage, that is, external storage outside the FPGA chip, which is represented by DDR on the FPGA development board.

[0052] Example 2

[0053] This embodiment further discloses the following contents based on the embodiment 1:

[0054] When the DCNN model includes a pooling layer, a pooling processing module is connected between each convolution processing module, and the pooling processing module processes the pooling layer part in the deep convolutional neural network.

[0055] When the parameters of the pooling layer have PKi =2 and Ps i =2, in the pooling module, the output of each pooling buffer is connected to a comparator Comp, and the three Comps form a 4-input comparator tree. The output of the comparator tree is connected to CCM i+1 In addition, the parallelism of the comparator tree is Hin i+1 The pooling module sends data to the CCM in parallel. i+1 R i+1 row data buffer outputs data, so the output data dimension of the pooling module is r i+1 ×Hin i+1 .

[0056] Example 3

[0057] This embodiment, based on Embodiment 1 or Embodiment 2, further discloses the following contents:

[0058] Assume that the number of convolutional layers of the convolutional neural network is L, and the width and height of the input feature map (Ifmap) corresponding to each convolutional layer are Hin i , the number of input channels is C i , the number of output channels is N i , the width and height of the output feature map (Ofmap) are both Hout i , the size of the convolution kernel corresponding to Ifmap is N i ×C i ×K i ×K i , stride is S i , padding is pad i Without loss of generality, we assume that there is a pooling layer (PL i ), the size of the pooling filter is PK i , the stride of the pool is Ps i , where all 1≤i≤L.

[0059] The layer-by-layer pipeline DCNN accelerator contains multiple convolution processing modules, whose number is equal to the number of convolution layers L of the DCNN model. The overall framework is shown in the attached Figure 1 The number of input rows of data for each convolution processing module is r i , the number of output rows is r i+1 The number of input rows of the convolution processing module is equal to the number of output rows of the previous convolution layer processing module. Figure 2, the external data source inputs a row of input feature map data to the accelerator every ΔT1 clock cycle. Every r2S1ΔT1 clock cycle, the first CCM outputs r2 rows of data to the next CCM. Similarly, every r i+1 S i ΔT i Clock cycles, CCM i Complete i+1 The calculation of Ofmap data is as follows:

[0060]

[0061] In the PE array, each computing module (PE module) is a multiplication-accumulation tree structure, see the attached Figure 4 , where W h，i and I w，i is the height and width of the PE array, W w，i is the number of input ports of the multiplication and accumulation tree. The intermediate calculation data is cached in the Dualport RAM, and the final calculation result is cached in the RAM. Among them, the RAM serves as the data source of the data buffer of the next CCM. Each PE array has a matching PE buffer with a Ping-Pong structure. The PE buffer is used to cache the input data of the PE array. The data comes from the data buffer of the convolution processing module. The data buffer is used to cache the output data from the previous convolution processing module. The data stored in each data buffer has the same row and column position in the input feature map, and is arranged in the data buffer according to the channel size. See the attached Figure 5 , the data stored in each data buffer has the same row position and column position in the input feature map, and is arranged in the data buffer according to the channel size.

[0062] Furthermore, the number of data buffers is #RowDataBuffer×Hin i .

[0063]

[0064]

[0065] DataIn0′=K i +S i (r i+1 -1)+GCD(r i+1 S i , r i )(r i ′-1)

[0066] Among them, GCD(r i+1 S i , r i ) is r i and r i+1 S i The greatest common divisor, r i ′ and (r i+1 S i )′ are two positive integers that are relatively prime.

[0067] Furthermore, the computational paradigm of the PE array is based on Toeplitz matrix multiplication, that is, the input feature map data needs to be converted into a Toeplitz matrix. The input feature map data processed by each PE array is located in the column matrix of the Toeplitz matrix, see Appendix Figure 6 , is a convolution operation based on the Toeplitz matrix. Each PE array performs small-size matrix multiplication W in sequence. rb ×I rpb , thus finally achieving large-size matrix multiplication W r ×I rp , the final calculation result is a single row of output feature map data.

[0068] The order of PE array calculation of weight data of single-row output feature map data and Toeplitz matrix data is shown in the attached Figure 7 In the hybrid multi-row data flow strategy, all parallel PE arrays share the same weight parameters to process the Ifmap data from different column matrices in the Toeplitz matrix respectively, thereby optimizing bandwidth usage. Figure 7 Shown is an example of two parallel PE array operating modes.

[0069] Furthermore, the computing resource usage of each PE array is

[0070] #PE Mult=W h，i ×W w，i ×I w，i

[0071] Among them, W h，i , W w，i , I w，i Needs to be satisfied

[0072]

[0073] Among them, ΔT1 is the clock cycle interval between the external data source inputting row-by-row data to the accelerator, satisfying

[0074]

[0075] Among them, TRP obj The expected throughput designed for the accelerator. The total computational effort of the DCNN model executed by the accelerator is set to IOP. freq is the operating frequency of the accelerator.

[0076] Furthermore, when the pooling module responsible for processing the pooling operation is used to process the output data of the convolution processing module, the number of its input row data is equal to the number of output row data of the convolution processing module, and the number of data buffers included satisfies

[0077] Hout i ×#RowPoolingBuffer

[0078] in

[0079]

[0080] Furthermore, the implementation of the fully connected processing unit includes a PE array and a data access system. Specifically, the PE array consists of batches of PEs. All PEs share the same weight data, but process different Ifmap data, that is, each PE is responsible for processing Ifmap data from different inference tasks. The hardware structure of PE is a MACC_f-input multiplication-accumulation tree, so each PE can implement MACC_f MACC operations in each clock cycle.

[0081] The same or similar reference numerals correspond to the same or similar components;

[0082] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0083] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A DCNN accelerator based on a hybrid multi-line data flow strategy, characterized in that: It includes a convolution processing unit and a fully connected processing unit, where: The convolution processing unit is responsible for processing the convolution calculation part in the DCNN model. The convolution processing unit includes multiple convolution processing modules, bypass convolution processing modules and branch processing modules. The multiple convolution processing modules are connected in sequence. The number of convolution processing modules is equal to the number of convolution layers L of the DCNN model. The number of input row data of each convolution processing module is r. i , the number of output rows is r i+1 , and the number of input row data of the convolution processing module is the number of output row data of the previous convolution processing unit; the number of input row data of the bypass convolution processing module is the number of output row data of the first convolution processing module, the output of the bypass convolution processing module is the input of the branch processing module, the output of the branch processing module is the input of the last convolution processing module, and the branch processing module processes the branch part in the deep convolutional neural network; The output row data of the last convolution processing module is output to the fully connected processing unit, which is used to process the fully connected layer part in the deep convolutional neural network; A pooling processing module is connected between each convolution processing module, and the pooling processing module processes the pooling layer part in the deep convolutional neural network; The external data source inputs a row of input feature map data to the first convolution processing module every ΔT1 clock cycle, and every r i+1 S i ΔT i clock cycles, the i-th convolution processing module completes r i+1 The calculation of the row output feature map data, where Where Ps j is the stride of the pooling module after the jth convolutional processing module, S j is the stride of the input feature map corresponding to the jth convolution processing module, and the size of the convolution kernel corresponding to the input feature map is N i ×C i ×K i ×K i , padding is pad i .

2. The DCNN accelerator based on the hybrid multi-row data flow strategy according to claim 1, characterized in that: Each of the convolution processing modules includes an input data buffer, a plurality of parallel computing buffers, a plurality of parallel computing unit arrays, and an output data buffer, wherein: The input data buffer reads and saves the data in the output data buffer of the previous convolution processing module, the multiple parallel computing buffers read the data in the input data buffer, the input of the multiple parallel computing unit arrays is the data in the computing buffer, and the output of the multiple parallel computing unit arrays is saved to the output data buffer.

3. The DCNN accelerator based on the hybrid multi-row data flow strategy according to claim 2, characterized in that: The plurality of parallel computing unit arrays are composed of W h，i ×I w，i computing units, each of which is a W w，i The input multiplication-accumulation tree, the intermediate calculation data is cached in the dual port RAM, and the final calculation result is cached in the RAM, where the RAM serves as the data source of the input data buffer of the next convolution processing module.

4. The DCNN accelerator based on the hybrid multi-row data flow strategy according to claim 3, characterized in that: Each computation unit performs small-size matrix multiplications W in sequence. rb ×I rpb , thus finally achieving large-size matrix multiplication W r ×I rp , the final calculation result is a single row of output feature map data.

5. The DCNN accelerator based on hybrid multi-row data flow strategy according to claim 4, characterized in that: The computing paradigm of each computing unit is based on Toeplitz matrix multiplication, which converts the input feature map data into a Toeplitz matrix. The input feature map data processed by each PE array is located in the column matrix of the Toeplitz matrix. In the mixed multi-row data flow strategy, all parallel computing unit arrays share the same weight parameters to separately process the Ifmap data from different column matrices in the Toeplitz matrix, thereby optimizing bandwidth usage.

6. The DCNN accelerator based on hybrid multi-row data flow strategy according to claim 5, characterized in that: The computing resource usage #PE Mult of each computing unit is: #PE Much=W h，i ×W w，i ×I w，i Among them, W h，i , W w，i , I w，i Needs to be satisfied Where Hout i is the width and height of the output feature map corresponding to the i-th convolution processing module, and ΔT1 is the clock cycle interval for the external data source to input rows of data to the accelerator, satisfying:

7. The DCNN accelerator based on hybrid multi-row data flow strategy according to claim 6, characterized in that: The data stored in each input data buffer has the same row and column positions in the input feature map and is arranged in the input data buffer according to the channel size. The number of data buffers in the i-th convolution processing module is #RowDataBuffer×Hin i : DataIn0′=K i +S i (r i+1 -1)+GCD(r i+1 S i ,r i )(r i ′-1) Among them, GCD(r i+1 S i , r i ) is r i and r i+1 S i The greatest common divisor of i ′ and (r i+1 S i )′ are two positive integers that are mutually prime, padding is the step size corresponding to the output feature map, and padding is pad i .

8. The DCNN accelerator based on hybrid multi-row data flow strategy according to claim 7, characterized in that: When the pooling processing module responsible for processing the pooling operation is used to process the output data of the convolution processing module, the number of its input row data is equal to the number of output row data of the convolution processing module, and the number of data buffers included satisfies: Wood i ×#RowPoolingBuffer in,

Images