Hardware architecture and design space exploration method for accelerating multi-channel convolution

Abstract

The application discloses a hardware architecture and a design space exploration method for accelerating multi-channel convolution, comprising an external storage module, a preprocessing module, a cache module, a matrix multiplication module and an output dimension rearrangement module, the external storage module is connected to the preprocessing module, the preprocessing module is connected to the cache module, the cache module is connected to the matrix multiplication module, the matrix processing module is connected to the output dimension rearrangement module; a task level pipeline is formed among the preprocessing module, the cache module and the matrix multiplication module, and parallel processing of three tasks of matrixing of a feature map and a convolution kernel, intermediate value caching and matrix multiplication operation is realized. The application converts multi-channel convolution into matrix multiplication and accelerates the same through design of a hardware structure, simultaneously proposes a design space exploration method based on performance and execution time, maximally improves resource utilization, and efficiently realizes real-time calculation of multi-channel convolution.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of convolutional neural network accelerator, and particularly relates to a hardware architecture for accelerating multi-channel convolution and a design space exploration method. BACKGROUND

[0002] Convolutional neural networks have made great progress in the field of computer vision and play an important role in processing perception tasks, and are widely used in information physical systems such as autonomous driving, robot systems and factory automation. However, the convolution operation is a complex calculation process for processing a large amount of data, which will bring a large time overhead and is the most time-consuming part of the convolutional neural network. Therefore, a hardware architecture needs to be designed to accelerate the multi-channel convolution operation.

[0003] The current research on hardware architecture for accelerating multi-channel convolution is still in exploration, and the current hardware architecture has the problems of unbalanced resource utilization and inability to maximize the use of resources. For example, the FINN architecture opened and restructured by Xilinx has a block random memory (BRAM) utilization rate of 87% and a flip-flop utilization rate of only 27% in terms of resource utilization. The inability to fully utilize resources also limits the overall throughput of the system and increases the processing delay. This field still has great research value and development space. SUMMARY

[0004] The purpose of the application is to provide a hardware architecture for accelerating multi-channel convolution and a design space exploration method, which solves the problem of unbalanced resource utilization and inability to maximize the use of resources in the hardware design architecture, and provides a new technical method for accelerating the processing of multi-channel convolution.

[0005] In order to achieve the above purpose, the technical scheme of the application is:

[0006] A hardware architecture for accelerating multi-channel convolution, comprising a cache module, further comprising an external storage module, a preprocessing module, a matrix multiplication module and an output dimension rearrangement module, the external storage module is connected to the preprocessing module, the preprocessing module is connected to the cache module, the cache module is connected to the matrix multiplication module, and the matrix processing module is connected to the output dimension rearrangement module; the preprocessing module, the cache module and the matrix multiplication module form a task-level pipeline, realizing parallel processing of three tasks of matrixing of feature maps and convolution kernels, intermediate value caching and matrix multiplication operation.

[0007] Further, the preprocessing module is composed of a plurality of processing unit arrays, and the processing units are connected to each other, and each processing unit processes a part of the input feature map and the multi-channel convolution kernel.

[0008] Further, the cache module is composed of block random memory (BRAM) on chip, and the result after preprocessing is cached.

[0009] Further, the matrix multiplication module is implemented by using systolic array, and has the single instruction multiple data (SIMD) characteristic; the row processing array number of the systolic array corresponds to the row dimension of the matrix A, the column processing array number corresponds to the column dimension of the matrix B, and there is a data channel between the processing units, and one processing unit transmits data to the processing unit on the right side and the processing unit below.

[0010] A design space exploration method of a hardware architecture for multi-channel convolution, comprising the following steps:

[0011] Step one: according to different board card models, set the FPGA resource model constraint condition;

[0012] Step two: set the parallelism constraint condition of the hardware architecture, and the single instruction multiple data parallelism of the matrix multiplication module in the hardware architecture cannot exceed the dimension of the matrix;

[0013] Step three: set the algorithm execution time constraint condition of the hardware architecture, divide the execution process of the hardware architecture into two basic meta-operations: memory access and calculation, determine the execution time of the two meta-operations on the basic components, and determine the execution time constraint related to the parallelism;

[0014] Step four: under the current resource constraint and algorithm execution time constraint condition, explore the parallelism of the task-level pipeline and the parallelism of the processing unit of the matrix processing module by using the enumeration method, so as to determine the parallelism of the task-level pipeline and the parallelism of the processing unit of the matrix processing module under the constraint conditions.

[0015] Compared with the prior art, the beneficial effects of the present application are:

[0016] 1. The present application designs a hardware structure to convert multi-channel convolution into matrix multiplication and accelerate it, maximizes the resource utilization rate, and can efficiently realize real-time calculation of multi-channel convolution.

[0017] 2. The matrix multiplication module proposed in the present application method supports single instruction multiple data (SIMD) parallel processing, supports setting different row and column parallelism, and performs multiplication of multiple row and column elements at a time. At the same time, there is a data channel between the processing units, and data can flow between the processing units, thereby reducing the delay caused by long-distance distribution of data.

[0018] 3、The method of the present application forms a task-level pipeline between the preprocessing module, the cache module and the matrix multiplication module, realizes parallel processing of the three tasks of matrixing of feature maps and convolution kernels, intermediate value caching and matrix multiplication operation, solves the blocking problem caused by input and output data stop-and-wait in the original serial processing flow by parallel processing of multiple tasks, spreads the time overhead of the preprocessing module task, shortens the time interval of the entire execution process each time, and thus improves the real-time performance of the execution process.

[0019] 4、The design space exploration method of the present application can set the FPGA resource model constraint according to different board card selection, divide the execution process of the hardware architecture into two basic meta-operations: memory access and calculation, determine the execution time of the two meta-operations on the basic components, determine the execution time constraint related to the parallelism degree, adjust the constraint condition of the resource, and explore the parallelism degree design scheme that meets the performance and algorithm execution time requirement of the application scene requirement.

[0020] 5、The present application explores the design space of the hardware architecture according to the resource constraint and the algorithm execution time constraint in the embedded application scene with limited computing resources and storage resources, determines the optimal design scheme, and deploys it on the FPGA platform to realize the acceleration processing of multi-channel convolution. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The hardware architecture used in the embodiments of the present application is shown in the figure;

[0022] Figure 2 The preprocessing module method used in the embodiments of the present application is shown in the figure;

[0023] Figure 3 The matrix multiplication acceleration module method used in the embodiments of the present application is shown in the figure;

[0024] Figure 4 The design space exploration method used in the embodiments of the present application is shown in the figure; DETAILED DESCRIPTION

[0025] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0026] A hardware framework for accelerating multi-channel convolution is designed and implemented on a xilinx Zynq UltraScale+MPSoC heterogeneous processor chip. Figure 1 As shown in the figure, it includes five parts of external storage module, preprocessing module, cache module, matrix multiplication module and output dimension rearrangement module. The external storage module is connected to the preprocessing module, the preprocessing module is connected to the cache module, the cache module is connected to the matrix multiplication module, and the matrix multiplication module is connected to the output dimension rearrangement module. The preprocessing module, cache module and matrix multiplication module form a task-level pipeline, realizing parallel processing of three tasks of feature map and convolution kernel matrix, intermediate value cache and matrix multiplication operation. The specific description is as follows:

[0027] The external storage module is composed of 4GB double rate synchronous random access memory (DDR), which is used to store the input feature map of all convolution layers waiting for processing and the multi-channel convolution kernel, or the output feature map output by the output dimension rearrangement module is also stored in this module.

[0028] The preprocessing module is composed of a plurality of processing unit arrays, which are connected to each other. Each processing unit processes a part of the input feature map and multi-channel convolution kernel.

[0029] The cache module is composed of block random access memory (BRAM) on the chip, which mainly caches the results after the preprocessing module processing for the matrix multiplication module to operate and process.

[0030] The matrix multiplication module is implemented by using systolic array, which has the characteristics of single instruction multiple data (SIMD). The number of row processing arrays of the systolic array corresponds to the row dimension of matrix A, and the number of column processing arrays corresponds to the column dimension of matrix B. There is a data path between the processing units. A processing unit transmits data to the processing unit on its right side and the processing unit below it, which is used to accelerate matrix multiplication.

[0031] The output dimension rearrangement module is used to rearrange the dimension of the result matrix obtained by matrix multiplication. The two-dimensional matrix form is rearranged according to the three dimensions of output channel, output feature map length and output feature map width, and output in the form of three-dimensional array.

[0032] The specific processing process of the preprocessing module is as shown in Figure 2 As shown in Figure 2 , the convolution kernel slides from the top of the single-channel input feature map, and the elements in the sliding window are stretched into a one-dimensional column vector. The convolution kernel keeps sliding on the input feature map to complete the above operation in turn until the convolution kernel slides to the bottom of the input feature map. This completes the matrixization processing of the single-channel input feature map. Then, this operation is repeated constantly to complete the matrixization processing of all input channels of the input feature map in turn. In this preprocessing process, the dimensions of the data are converted. The dimensions of the input data are represented in the form of a three-tuple as <input feature map channel number, input feature map width, input feature map height>, and the dimensions of the output data are represented in the form of a two-tuple as <convolution kernel width * convolution kernel height * input feature map channel number, output feature map width * output feature map height>. In the matrixization processing of the convolution kernel, the single-channel convolution kernel is stretched into a one-dimensional row vector, which completes the matrixization processing of the single-channel convolution kernel. Then, this operation is repeated constantly to complete the matrixization processing of all channels of the convolution kernel in turn, and these row vectors are spliced together. Similarly, in this process, there is also a conversion of the dimensions of the data. The dimensions of the input data are represented in the form of a three-tuple as <convolution kernel channel number, convolution kernel width, convolution kernel height>, and the dimensions of the output data are represented in the form of a two-tuple as <convolution kernel width * convolution kernel height * input feature map channel number, output feature map channel number>.

[0033] The specific processing of the matrix multiplication module is as shown in Figure 3 , the processing module is implemented based on the systolic array structure. The number of row processing arrays of the systolic array corresponds to the row dimension of matrix A, and the number of column processing arrays corresponds to the column dimension of matrix B. In the first clock cycle, the first unit of the processing array is activated, which receives input data from the buffer blocks above and to the left, and performs operations. In the second clock cycle, the units to the right and below the first processing unit are activated, and the first unit continues to receive input data from the buffer blocks above and to the left, while sending the data received in the previous clock cycle to the right and below units. These activated units receive input data and perform operations. This process is repeated constantly until the last unit of the processing array completes the operation.

[0034] As shown in Figure 4 , a design space exploration method based on the hardware architecture for multi-channel convolution described above, specifically comprising the following steps:

[0035] Step 1: According to different board models, set the FPGA resource model constraint condition. The DSP resources and BRAM resources used by the hardware architecture cannot exceed the constraint value.

[0036] Step two: setting the hardware architecture parallelism constraint condition, the single instruction multiple data parallelism of the matrix multiplication module in the hardware architecture cannot exceed the dimension of the matrix;

[0037] Step three: setting the algorithm execution time constraint condition of the hardware architecture, dividing the execution process of the hardware architecture into two basic meta operations: memory access and calculation, determining the execution time of the two meta operations on the basic components, and determining the execution time constraint related to the parallelism;

[0038] Step four: under the current resource constraint and the algorithm execution time constraint condition, exploring the parallelism of the task-level pipelining and the parallelism of the processing unit of the matrix processing module using the enumeration method, so as to determine the parallelism of the task-level pipelining and the parallelism of the processing unit of the matrix processing module under the constraint conditions.

[0039] The above application uses specific examples to illustrate the present application, which is only used to help understand the present application and does not limit the present application. Any modification or replacement of the skilled person within the technical scope disclosed by the present application should be covered within the scope of the present application.

Claims

1. A hardware architecture for accelerating multi-channel convolution, comprising a cache module, characterized in that: Further comprising an external storage module, a preprocessing module, a matrix multiplication module and an output dimension rearrangement module, the external storage module is connected to the preprocessing module, the preprocessing module is connected to the cache module, the cache module is connected to the matrix multiplication module, the matrix multiplication module is connected to the output dimension rearrangement module; the preprocessing module, the cache module and the matrix multiplication module form a task-level pipeline, realizing parallel processing of three tasks of matrixing of feature maps and convolution kernels, intermediate value caching and matrix multiplication operation. The matrix multiplication module is implemented by using a systolic array, and has the characteristics of single instruction multiple data (SIMD). The number of row processing arrays of the systolic array corresponds to the row dimension of matrix A, and the number of column processing arrays corresponds to the column dimension of matrix B. There is a data path between the processing units, and one processing unit transmits data to the processing unit on its right side and the processing unit below it.

2. The hardware architecture for accelerating multi-channel convolution of claim 1, wherein: The preprocessing module is composed of a plurality of processing unit arrays, and the processing units are connected to each other. Each processing unit processes a part of the input feature map and the multi-channel convolution kernel.

3. The hardware architecture for accelerating multi-channel convolution of claim 2, wherein: The cache module is composed of block random access memory (BRAM) on the chip, and caches the results after preprocessing.

4. The design space exploration method for a hardware architecture of multi-channel convolution of claim 1, wherein, The method comprises the following steps: Step one: set the FPGA resource model constraint condition according to different board card models; Step two: set the hardware architecture parallelism constraint condition. The single instruction multiple data parallelism of the matrix multiplication module in the hardware architecture cannot exceed the dimension of the matrix; Step three: set the algorithm execution time constraint condition of the hardware architecture. The execution process of the hardware architecture is divided into two basic meta-operations: memory access and calculation. The execution time of the two meta-operations on the basic components is determined, and the execution time constraint related to the parallelism is determined; Step four: under the current resource constraint and algorithm execution time constraint condition, use the enumeration method to explore the parallelism of the task-level pipeline and the parallelism of the processing unit of the matrix multiplication module, so as to determine the parallelism of the task-level pipeline and the parallelism of the processing unit of the matrix multiplication module under the constraint conditions.

Images