A high-parallelism heterogeneous convolutional neural network accelerator and acceleration method

Abstract

The application discloses a high-parallelism heterogeneous convolutional neural network accelerator and an acceleration method, which comprises a control subsystem, a parallel processing subsystem and a storage subsystem, a control module in the subsystem is used for loading input data from an off-chip storage module to an on-chip storage module or writing output data from the on-chip storage module back to the off-chip storage module, the on-chip storage module is used for buffering input data to be processed or operation results obtained by a parallel computing module, a configurable weight quantization module is used for reading weights from the on-chip storage module and quantizing according to a configuration file, and an instruction module obtains instructions from the off-chip storage module and drives the parallel computing module to perform operation; and the parallel processing module is used for processing input data and weights, and realizes convolution operation or pooling operation according to the instructions. The application fully utilizes control and logical computing resources of a heterogeneous processor, realizes higher-parallelism data processing, and enables the system to realize a convolutional neural network with lower power consumption and higher performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of convolutional neural network accelerator, and particularly relates to a high-parallelism heterogeneous convolutional neural network accelerator and an acceleration method. BACKGROUND

[0002] The convolutional neural network has made great progress in the field of computer vision, and has been widely applied in image classification, target detection, security monitoring video analysis and other scenes. Compared with traditional methods, the convolutional neural network relies on deeper network layers and more model parameters to obtain strong ability to learn and extract features from a large amount of data, and has greatly exceeded the accuracy of human beings in many visual tasks.

[0003] With the increasing richness of the landing business of the convolutional neural network, high requirements are put forward for the real-time processing and the power consumption of the device. At present, the computing platform of the convolutional neural network mainly includes CPU, GPU and FPGA, the CPU is good at processing the serial control flow, and has certain disadvantages in processing the large-scale parallel neuron calculation of the neural network, the GPU has certain parallelism, but the power consumption is high, which cannot meet the requirements of the landing business on the power consumption, the FPGA has strong parallel processing capability, is suitable for processing a large amount of parallel operation, and has the characteristics of dynamic configuration and low running power consumption, but has certain disadvantages in task control. How to give full play to the advantages of multiple computing platforms, design the convolutional neural network accelerator based on the heterogeneous processor, improve the real-time processing and reduce the power consumption, has attracted more and more attention of researchers. The current domestic and foreign research on the heterogeneous convolutional neural network accelerator is still in continuous exploration, the structure of the current heterogeneous convolutional neural network accelerator includes the DPU acceleration engine based on the zynq platform and the TPU accelerator adopting the systolic array structure, although certain achievements have been made, but it is still far from mature, the current design scheme has low calculation parallelism, cannot fully utilize the computing resources, and in the edge computing scene, there are problems of high power consumption and poor real-time performance in the execution process of the convolutional neural network accelerator, and the field still has great research value and development space. SUMMARY

[0004] The application aims to provide a high-parallelism heterogeneous convolutional neural network accelerator and an acceleration method, solve the technical problems of high power consumption and poor real-time performance of the embedded system in the convolutional neural network inference process in the edge computing scene, and provide a new technical method for accelerating the processing of the convolutional neural network.

[0005] In order to achieve the above object, the technical scheme of the present application is: a high-parallelism heterogeneous convolutional neural network accelerator, comprising a control subsystem, a parallel processing subsystem and a storage subsystem, the control subsystem being connected to the parallel processing subsystem, the control subsystem being connected to the storage subsystem, and the storage subsystem being connected to the parallel processing subsystem; the control subsystem comprising a control module, an instruction module and a configurable weight quantization module; the processing subsystem comprising a convolution parallel computation module and a pooling module; and the storage subsystem comprising an off-chip storage module, an on-chip storage module.

[0006] Further, the convolution computation module is composed of a multi-level processing unit array, the processing unit array of the first level being responsible for parallel processing of input channels, and the processing unit of the second level being responsible for parallel processing of output channels; the processing procedure is realized in a pipeline manner, and the three processes of reading data, operation and writing back data are realized in a multi-stage pipeline.

[0007] Further, the on-chip storage module is composed of a buffer storage level, a cache storage level and a first-in-first-out storage level.

[0008] Further, the acceleration method of the high-parallelism heterogeneous convolutional neural network accelerator comprises the following steps:

[0009] Step one: in the control subsystem, the control module initializes the off-chip storage module, writes the corresponding configuration file, network structure file, input picture and weight data into the off-chip storage module;

[0010] Step two: the control module drives the instruction module, the instruction module reads the network structure file in the off-chip storage module, automatically parses the file, generates corresponding configuration instructions and scheduling instructions, and sends the configuration instructions to the configurable weight quantization module and the scheduling instructions to the parallel processing subsystem;

[0011] Step three: the configurable weight quantization module reads the configuration file in the off-chip storage module, calls a quantization program according to the configuration file, performs corresponding quantization on the weight data, and writes the quantized weight data back to the off-chip storage module;

[0012] Step four: the on-chip storage module reads the input picture and weight data from the off-chip storage module, and completes the caching of the current row data and the adjacent two rows of data, and loads the corresponding data into the corresponding FIFO block;

[0013] Step five: the parallel processing subsystem parses the scheduling instructions, and executes the convolution computation module and the pooling computation module in sequence according to the order of the network structure;

[0014] Step six: when the input feature map and the output feature map are unfolded by the convolution calculation module, the parallel degree is set as Tm and Tn, Tm input feature maps and Tn output feature maps are unfolded at a time, and Tm*Tn groups of vectors are processed in parallel; when the convolution calculation module calculates, the multi-channel input feature map is unfolded along the input channel and the output channel, and the multiplication and accumulation operation is performed on the data in the two dimensions in parallel;

[0015] Step seven: the pooling calculation module takes the calculation results of the convolution calculation module as input, performs multiple comparison operations on the results, obtains the final calculation result, and writes the result back to the off-chip storage module through the on-chip storage module.

[0016] Compared with the prior art, the method has the following beneficial effects:

[0017] 1. The parallel calculation module of the method solves the read-write dependence problem of the convolution operation in the traditional accelerator calculation design scheme, adjusts the read-write access of two adjacent cycles to the same block of memory addresses to read-write access to different blocks of memory addresses, can completely pipeline the convolution operation process, increases the utilization rate of processor calculation resources and logical resources, and thus reduces the running time.

[0018] 2. The on-chip storage module of the method, through a multi-level storage structure, caches the data of the current row and the adjacent two rows, improves the data reuse rate, and reduces the number of access to the memory; by designing the FIFO cache level corresponding to the parallel calculation unit, each FIFO block only improves the data reading and writing back function for the corresponding processing unit array, realizes parallel loading of data to the operation unit, and thus realizes higher parallel degree of data processing.

[0019] 3. The accelerator of the method supports dynamic bit width quantization of weight data, so that the accelerator is suitable for operation and processing of weight data with multiple bit widths.

[0020] 4. The accelerator of the method has programmable and dynamically reconfigurable functions, different configuration commands are set in the command module of the control subsystem to realize acceleration of multiple network structures; and the input, output channel, size and dimension of the convolution kernel, and the parallel degree of calculation also support dynamic programming and configuration.

[0021] 5. The accelerator of the method realizes a hardware-optimized network structure, which adopts the implementation mode of performing pooling first and then performing activation, which reduces the size of the input feature map sent to the activation function through the pooling function, thereby reducing the operation amount of the network structure. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1The overall architecture schematic diagram of the convolutional neural network accelerator used in the embodiment of the present application is shown in the figure;

[0023] Figure 2 The configurable weight quantization module method used in the embodiment of the present application is shown in the figure;

[0024] Figure 3 The on-chip storage module method used in the embodiment of the present application is shown in the figure;

[0025] Figure 4 The parallel computing subsystem method used in the embodiment of the present application is shown in the figure;

[0026] Figure 5 The convolutional computing module method used in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0028] A high-parallelism heterogeneous convolutional neural network accelerator is designed and implemented on a xilinx Zynq UltraScale+MPSoC heterogeneous processor chip. As shown in the figure, Figure 1 A high-parallelism heterogeneous convolutional neural network accelerator includes a control subsystem, a parallel processing subsystem and a storage subsystem. The control subsystem is connected to the parallel processing subsystem, the control subsystem is connected to the storage subsystem, and the storage subsystem is connected to the parallel processing subsystem. The control subsystem includes a control module, an instruction module and a configurable weight quantization module. The processing subsystem includes a convolutional parallel computing module and a pooling module. The storage subsystem includes an off-chip storage module and an on-chip storage module.

[0029] The control module is used to load input data from the off-chip storage module to the on-chip storage module, or write output data from the on-chip storage module back to the off-chip storage module. The on-chip storage module is used to cache input data to be processed or operation results obtained by the parallel computing module. The configurable weight quantization module reads weights from the on-chip storage module and quantizes according to the configuration file. The instruction module mainly obtains instructions from the off-chip storage module and drives the parallel computing module to perform operations. The parallel processing subsystem is used to process input data and weights, and realizes convolutional computing module or pooling computing module operations according to instructions.

[0030] Specifically, in the control subsystem,

[0031] The control module is implemented based on an ARM CPU processor and a Linux operating system. By deploying a corresponding control system, it configures the instruction module and quantization module, and schedules the storage and computing modules. The interface between the control system and the external environment is determined by the on-chip bus protocol. The instruction transmission interface uses the AXI-Lite interface, and the transmission interface between the on-chip storage module, the off-chip storage module, and the parallel computing module uses the AXI interface. The function of the control module is to load input data from the off-chip storage module into the on-chip storage module, or to write data from the off-chip storage module back to the on-chip storage module.

[0032] The instruction module primarily retrieves instructions from external memory and drives the parallel computing module to execute corresponding operations, or generates configuration files with different quantization bit widths. The instruction module is responsible for initializing the configurable weight quantization module and initializing and scheduling the execution of the parallel computing module; it reads the network structure and configuration parameters stored in external memory, and initializes the parameter configuration, performs data scheduling, and computation scheduling for the parallel computing module. The configurable weight quantization module reads weights from the on-chip storage module and quantizes the read weights according to the configuration file; it supports multiple quantization bit widths and quantizes weight values ​​according to the configuration file.

[0033] The parallel processing subsystem includes:

[0034] The convolutional computation module unfolds the multi-channel input feature map along two dimensions: input channel and output channel. It performs multiplication and accumulation operations on the data in these two dimensions in parallel. It consists of a multi-level processing unit array. The first-level processing unit array is responsible for the parallel processing of the input channel, and the second-level processing unit array is responsible for the parallel processing of the output channel. The processing is implemented in a pipeline manner, realizing multi-level pipelined processing of the three processes of reading data, computing, and writing back data.

[0035] Convolutional neural network models mainly consist of convolutional layers, pooling layers, and fully connected layers. The primary implementation of convolutional layers is matrix multiplication and addition, and the computation of fully connected layers can also be abstracted as matrix multiplication and addition. During computation, the operations of these two types of layers are mapped to the same operation, which is accomplished by reusing the convolutional computation module.

[0036] The convolution calculation module is composed of a multiplier, an adder and an intermediate result register, mainly responsible for the multiplication and accumulation operation of the matrix, each multiplication and accumulation calculates multiple adjacent convolution sliding windows; the multiplication and accumulation operation of the data in the two dimensions is carried out in parallel by unfolding the two dimensions; the parallel calculation is realized by a multi-level processing unit array in the hardware structure, the first level processing unit array is responsible for the parallel processing of the input channel, and the second level processing unit is responsible for the parallel processing of the output channel, and the number of channels processed in parallel by the two levels each time is determined by the parallelism in the configuration file;

[0037] The pooling calculation module is composed of a comparator, which is used to realize the comparison operation of two numbers, and the output feature map obtained by the convolution calculation module is used as the input. The data in a sliding window is sent into the comparator for comparison to obtain the maximum value in the sliding window. The operation is repeated until the sliding window traverses the entire input, and the final operation result of the pooling is obtained, and the output is written back to the off-chip storage module.

[0038] Among the storage subsystem:

[0039] The on-chip storage module is composed of multiple storage levels: Buffer storage level, Cache storage level, FIFO storage level. The Buffer storage level is mainly used to store the input image data and the quantized weight data, the Cache storage level is mainly used to store the data of the current row and its adjacent two rows, and the FIFO storage level is mainly used to store the data in the sliding window being processed.

[0040] The off-chip storage module is composed of DRAM memory, mainly used to store the input data of the model, the weight parameters of the model, the network structure file and the configuration file.

[0041] A high-parallelism heterogeneous convolutional neural network accelerator acceleration method, comprising the following steps:

[0042] Step 1: refer to Figure 1 , in the control subsystem, the control module initializes the off-chip storage module, and stores the picture data, the network structure file of the convolutional neural network model, the input picture, the configuration file and the weight data into the sd card or the external storage DDR.

[0043] Step 2: the control module drives the instruction module, and the instruction module reads the network structure file in the external storage, automatically parses the file, and generates corresponding configuration instructions and scheduling instructions.

[0044] Step 3: the configurable weight quantization module reads the configuration file in the external storage, calls the quantization program according to the configuration file, and refers to Figure 2The weight data is quantized correspondingly, and the quantized weight data is written back to the external storage. The quantization method uses a quantization perception training method, which is implemented by training a model, parameter quantization and quantization calibration. The model is obtained by offline training on a cloud server. Then, the model parameters are quantized by a quantization tool, and the 32-bit floating-point numbers are converted into 16-bit fixed-point numbers. Finally, an unmarked calibration data set is used for precision checking to ensure that the precision is within an acceptable range.

[0045] Step 4: Refer to Figure 3 The input data and weight data are loaded from the off-chip storage module to the on-chip storage module, and the current row data and the adjacent two rows of data are buffered, and the corresponding data is recorded to the corresponding FIFO block.

[0046] Step 5: The parallel processing subsystem parses the scheduling instruction, completes the initialization configuration of the parallel computing module, determines the hardware template parameters of the convolution operation: the number of input channels, the number of output channels, the size of the convolution kernel, and the parallel degree; and executes the convolution calculation module and the pooling calculation module in sequence according to the order of the network structure.

[0047] Step 6: Refer to Figure 5 In the parallel processing subsystem, the convolution calculation module uses the following expansion method for the input feature map and the output feature map: the parallel degree is set to Tm and Tn, Tm input feature maps and Tn output feature maps are expanded at a time, and Tm*Tn groups of vectors are processed in parallel. Refer to Figure 4 The convolution operation of the convolution calculation module is to expand the multi-channel input feature map along the input channel and the output channel, and to perform multiplication and accumulation operation on the data of the two dimensions in parallel. The processing process includes the following three processes: loading the input picture data and weight data to be processed from the FIFO buffer block of the on-chip storage module, performing convolution operation to complete the multiplication and accumulation of the input picture and the weight, and writing the output feature map obtained by the final operation from the on-chip storage module to the off-chip storage module.

[0048] Step 7: The operation processing process of the pooling calculation module includes the following three processes: reading the output feature map from the external storage and loading it to the on-chip storage module, performing comparison operation or mean operation, and writing the final operation result to the off-chip storage module.

[0049] The above application uses specific examples to illustrate the present application, which is only used to help understand the present application and does not limit the present application. Any modification or replacement within the scope of the disclosed technology by those skilled in the art should be covered within the scope of the present application.

Claims

1. A method for accelerating a high-parallelism heterogeneous convolutional neural network accelerator, the method comprising: The method comprises the following steps: Step 1: in the control subsystem, the control module initializes the off-chip storage module, writes the corresponding configuration file, network structure file, input picture and weight data into the off-chip storage module; Step 2: the control module drives the instruction module, the instruction module reads the network structure file in the off-chip storage module, automatically parses the file, generates the corresponding configuration instruction and scheduling instruction, and sends the configuration instruction to the configurable weight quantization module and the scheduling instruction to the parallel processing subsystem; Step 3: the configurable weight quantization module reads the configuration file in the external storage, calls the quantization program according to the configuration file, performs corresponding quantization on the weight data, and writes the quantized weight data back to the off-chip storage module; Step 4: the on-chip storage module reads the input picture and weight data from the off-chip storage module, and completes the buffering of the current row data and the adjacent two rows of data, and loads the corresponding data into the corresponding FIFO block; Step 5: the parallel processing subsystem parses the scheduling instruction, and executes the convolution calculation module and the pooling calculation module in sequence according to the network structure; Step 6: when the convolution calculation module unfolds the input feature map and the output feature map, the parallel degree is set to Tm and Tn, Tm input feature maps and Tn output feature maps are unfolded at a time, and Tm*Tn groups of vectors are processed in parallel; when the convolution calculation module calculates, the multi-channel input feature map is unfolded along the input channel and the output channel, and the multiplication and accumulation operation is performed on the data of the two dimensions in parallel; Step 7: the pooling calculation module takes the calculation result of the convolution calculation module as input, performs multiple comparison operations on these results to obtain the final calculation result, and writes the result back to the off-chip storage module through the on-chip storage module. The accelerator used in the method comprises a control subsystem, a parallel processing subsystem and a storage subsystem, the control subsystem is connected to the parallel processing subsystem, the control subsystem is connected to the storage subsystem, and the storage subsystem is connected to the parallel processing subsystem; the control subsystem comprises a control module, an instruction module and a configurable weight quantization module; the processing subsystem comprises a convolution parallel calculation module and a pooling module; the storage subsystem comprises an off-chip storage module and an on-chip storage module.

2. The method of claim 1, wherein, The convolution calculation module is composed of a multi-level processing unit array, the first level processing unit array is responsible for parallel processing of the input channel, and the second level processing unit is responsible for parallel processing of the output channel; the processing process is realized based on the pipeline mode, and the reading data, operation and writing data processes are realized by multi-stage flow.

3. The method of claim 2, wherein, The on-chip storage module is composed of a buffer storage level, a cache storage level and a first-in-first-out storage level.

Images