A convolutional neural network accelerator based on FPGA

Abstract

The present invention discloses a convolutional neural network accelerator based on FPGA, including: a control unit for controlling the state conversion and data flow transmission process of the global calculation of the convolutional neural network; a cache unit for storing the weight data, input data, output data, and the calculation results of each layer of convolution and full connection required for calculation; a calculation unit for parallel calculation of the convolutional neural network; the calculation unit includes a convolution calculation unit and a full connection calculation unit; the convolution calculation unit includes N parallel calculation CPEs; the full connection calculation unit includes softmax and M parallel calculation FCPEs, and softmax is used to output the classification results of the last fully connected layer in a probabilistic manner. The present invention adopts a strategy of parallel calculation and reuse of multiple CPEs and FCPEs to improve the throughput of the accelerator, and adopts a multi-level block cache design to reduce a large amount of data movement during the calculation process, reduce the power consumption of the accelerator, so as to achieve the purpose of improving the energy efficiency of the accelerator.

Description

Technical Field

[0001] The present invention relates to the field of convolutional neural networks of artificial intelligence, and in particular to a convolutional neural network accelerator based on FPGA. Background Art

[0002] Convolutional neural network (CNN) is a typical deep learning network that extracts abstract features from raw data layer by layer through the stacking of multiple layers. CNN is a feedforward neural network that is mainly composed of input layer, convolution layer, pooling layer, fully connected layer and output layer. With the development of deep learning algorithms, from the initial small network LeNet-5 to the medium-sized Alexnet, VGG16, and even the large-scale ResNet network, the computational complexity and storage complexity of convolutional neural network algorithms are increasing. Therefore, how to implement high-performance and low-power convolutional neural network algorithms on existing hardware platforms has certain practical significance.

[0003] The hardware implementation platforms of convolutional neural network algorithms are mainly divided into four types: general-purpose processors CPU, graphics processors GPU, application-specific integrated circuits ASIC, and field-programmable gate arrays FPGA. CPUs are limited by serial execution mode and limited data bandwidth, and cannot meet the requirements of convolutional neural network operations in terms of power consumption and latency. Therefore, the hardware acceleration of convolutional neural networks is generally based on GPU, ASIC, and FPGA platforms. FPGA has the advantages of programmability, reconfigurability, high parallelism, and low power consumption. Compared with the fixed hardware structure of GPU and ASIC, FPGA can be designed to adapt to the hardware structure of various convolutional neural network algorithms. At the same time, compared with CPU and GPU, FPGA can obtain higher performance under unit power consumption, and has better energy consumption ratio and acceleration ratio. The existing FPGA-based convolutional neural network accelerator has low DSP performance efficiency, especially in the convolution calculation process, there is a large amount of data reuse, which is not reflected in the DSP calculation process. Therefore, it is a very worthy issue to study the high-performance accelerator of convolutional neural network based on FPGA according to CNN software algorithm, make full use of the advantages of FPGA operation, and achieve high acceleration ratio and high throughput with low power consumption. Summary of the invention

[0004] Purpose of the invention: In view of the above problems, the purpose of the present invention is to provide a convolutional neural network accelerator based on FPGA, which adopts the acceleration strategy of computational parallelism, pipeline design, data quantization, CPE and FCPE multiplexing, and reduces the FPGA on-chip storage overhead by selecting appropriate data paths, thereby improving the accelerator performance.

[0005] Technical solution: A convolutional neural network accelerator based on FPGA of the present invention comprises: a control unit for controlling the state conversion and data flow transmission process of the global calculation of the convolutional neural network; a cache unit for storing the weight data, input data, output data, and calculation results of each layer of convolution and full connection required for the calculation; a calculation unit for parallel calculation of the convolutional neural network;

[0006] The computing unit includes a convolution computing unit and a fully connected computing unit;

[0007] The convolution calculation unit includes N parallel convolution processing units CPE, each CPE includes a convolution module and a pooling activation module; the convolution module is composed of a DSP array, which is used to complete the convolution calculation and normalization operation; the pooling activation module is composed of a maximum comparator and a 0 value comparator, and the pooling activation module is used to perform maximum pooling and activation operations on the output result of the convolution module;

[0008] The fully connected computing unit includes softmax and M fully connected processing units FCPE for parallel computing. Each FCPE is composed of a multiplier and an adder. FCPE is used to perform multiplication and accumulation operations on input data and weight data. Softmax is composed of a divider, an adder and a maximum comparator. Softmax is used to output the classification results of the last fully connected layer in a probabilistic manner.

[0009] Furthermore, the calculation process of the convolution module includes the following steps:

[0010] Step 11, expand the two-dimensional matrix of input data and weight data into a one-dimensional matrix along the rows first and then the columns, and then convert the floating-point data of the one-dimensional matrix into corresponding fixed-point numbers;

[0011] Step 12, input the converted input data and weight data into the DSP array, and perform multiplication operation on the L input data and weight data in the convolution window first, and then perform accumulation operation between R channels;

[0012] The size of the DSP array is R×L, where R represents the number of channels for parallel computing and L represents the size of the convolution kernel;

[0013] Step 13, perform an addition operation on the L data accumulated between channels to obtain a convolution calculation result, and determine whether the convolution calculation is completed. If it is completed, execute step 14, otherwise return to step 12 to continue the calculation;

[0014] Step 14, normalize the convolution calculation result to obtain the output result of the convolution module, and cache the output result in the intermediate cache module of the cache unit.

[0015] Furthermore, the calculation process of the pooling activation module includes the following steps:

[0016] Step 21, performing maximum comparison on the pooling window row by row to obtain the maximum value of each row;

[0017] Step 22, perform maximum comparison on the maximum value of each row to obtain the maximum pooling result;

[0018] Step 23, sending the maximum pooling result as input data to the 0 value comparator, comparing the input data with 0, when the input data is less than 0, the output is 0; when the input data is greater than or equal to 0, the output data keeps the size of the input data unchanged, and caches the output data in the intermediate cache module of the cache unit.

[0019] Furthermore, the adder of FCPE adopts a five-stage pipeline adder, and each FCPE calculation process includes the following steps:

[0020] Step 31, sending the input data and weight data to each multiplier for multiplication operation at the same time to obtain each multiplication result;

[0021] Step 32, use a five-stage pipeline adder to accumulate all multiplication results and bias registers, and use the addition results to continuously update the value of the bias register. When all input data and weight data are multiplied and accumulated, the bias register stops updating, inputs the activation unit, obtains the output result of the fully connected layer, and caches the output result of the current layer in the intermediate cache module of the cache unit.

[0022] Furthermore, softmax includes exponential function operations. In hardware implementation, a lookup table is used to convert the exponential function with base e into an exponential function with base 1 / e through formula conversion. The converted exponential function is divided into a finite number of intervals. Each interval is approximated by a fixed coefficient value. The fixed coefficient value of each interval is stored in the BRAM of the FPGA. The softmax probability output includes the following steps:

[0023] Step 41, using the output data of the last fully connected layer as the input data of softmax, and finding the maximum value through the maximum comparator;

[0024] Step 42, calculating the difference between the maximum value and the input data, reading the fixed coefficient value of the interval corresponding to the difference from the BRAM, and accumulating;

[0025] Step 43, taking the fixed coefficient value of the interval corresponding to the difference as the numerator and the accumulated result as the denominator, and sending them to the divider for calculation to obtain the probability output result, and cache the output result in the output cache module of the cache unit.

[0026] Furthermore, the cache unit includes an off-chip DRAM, an input cache module, an output cache module, a weight cache module and an intermediate cache module. The off-chip DRAM is used to store off-chip data, the input cache module is used to store data read from the off-chip DRAM, the output cache module is used to store output data of the fully connected computing module, the weight cache module is used to store weight parameters, and the intermediate cache module is used to store the calculation results of each layer of convolution and the calculation results of each layer of full connection.

[0027] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0028] 1. The present invention adopts a strategy of parallel computing and reuse of multiple CPEs and FCPEs to improve the throughput of the accelerator, and adopts a multi-level block cache design to reduce a large amount of data movement during the calculation process, thereby reducing the power consumption of the accelerator, so as to achieve the purpose of improving the energy efficiency of the accelerator;

[0029] 2. The accelerator of the present invention has the characteristics of high energy efficiency compared with general-purpose processors;

[0030] 3. The DSP array structure of the present invention adopts a cascaded manner, has efficient data multiplexing, reduces data handling, and reduces accelerator power consumption overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is the hardware architecture diagram of the CNN accelerator based on FPGA;

[0032] Figure 2 It is the top-level state control flow chart of CNN;

[0033] Figure 3 It is the overall structure diagram of the convolution calculation unit;

[0034] Figure 4 It is the circuit structure diagram of the convolution module;

[0035] Figure 5 It is the circuit structure diagram of the pooling activation module;

[0036] Figure 6 It is the overall structure diagram of the fully connected computing unit;

[0037] Figure 7 It is the internal circuit structure diagram of FCPE;

[0038] Figure 8 This is the circuit structure diagram of softmax;

[0039] Fig. 9 This is a diagram of the accuracy of the first layer of convolution. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0041] like Figure 1 The FPGA-based convolutional neural network accelerator described in this embodiment includes: a control unit, a cache unit and a computing unit, and the computing unit includes a convolution computing unit and a fully connected computing unit.

[0042] The convolution calculation unit includes N parallel convolution processing units CPE, each CPE includes a convolution module and a pooling activation module. The convolution module is composed of a DSP array to complete the convolution calculation and normalization operation; the pooling activation module is composed of a maximum comparator and a 0 value comparator, and the pooling activation module is used to perform maximum pooling and activation operations on the output results of the convolution module.

[0043] The fully connected computing unit includes softmax and M parallel computing fully connected processing units FCPE. Each FCPE is composed of a multiplier and an adder. FCPE is used to perform multiplication and accumulation operations on input data and weight data. Softmax is composed of a divider, an adder and a maximum comparator. Softmax is used to output the classification results of the last fully connected layer in a probabilistic manner.

[0044] The cache unit includes an off-chip DRAM, an input cache module, an output cache module, a weight cache module and an intermediate cache module. The off-chip DRAM is used to store off-chip data, the input cache module is used to store data read from the off-chip DRAM, the output cache module is used to store the output data of the fully connected computing module, the weight cache module is used to store weight parameters, and the intermediate cache module is used to store the calculation results of each layer of convolution and each layer of full connection.

[0045] The control module provides the read and write related signals of the relevant cache modules and the control signals of the computing units, controls the state transition and data flow transmission process of the global calculation of the convolutional neural network, and keeps the accelerator calculation in an orderly manner.

[0046] In this embodiment, the convolution calculation unit in the CNN network includes two layers of convolution, two layers of normalization and two layers of pooling, and the fully connected calculation unit includes two layers of full connection and one layer of softmax. The specific network model parameters are shown in Table 1. In this embodiment, the number of convolution kernels in the convolution layer of the CNN network model is 16, and the convolution calculation unit is set with 16 CPEs, and the 16 CPEs complete the calculation of 16 convolution kernels in parallel; the first layer of full connection is the output of 128 nodes, and the second layer of full connection is the output of 6 nodes. The fully connected calculation unit is set with 8 FCPEs, and the first layer of full connection 8 FCPEs are calculated in parallel, and the output of 128 nodes is completed by multiplexing 16 times. There are two idle FCPEs in the second layer of full connection, and 6 FCPEs are calculated in parallel to complete the output of 6 nodes.

[0047] Table 1 CNN network model structure parameters

[0048]

[0049] When the network model changes and the number of convolution kernels changes, based on the design of 16 CPEs in this embodiment, the calculation can be completed by multiplexing in units of 16 or expanding in multiples of 16 while balancing FPGA resource utilization and accelerator computing performance. When the network model is different and the number of fully connected output nodes changes, based on the design of 8 FCPEs in this embodiment, the calculation can be completed by multiplexing in units of 8 or expanding in multiples of 8 while balancing FPGA resource utilization and accelerator computing performance.

[0050] The control unit is used to control the state transition and data flow transmission of the global calculation of the convolutional neural network, where the data flow transmission process includes: reading input data from the off-chip DRAM and writing it to the FPGA on-chip BRAM, and reading input data and weight data from the BRAM to the computing unit for calculation.

[0051] The control unit not only controls the intermediate data flow during the calculation of the accelerator computing unit, but also controls the overall calculation process of the convolutional neural network and reasonably allocates the switching between the various states of the accelerator. Figure 2As shown in the figure, the control unit consists of 5 states. S0 is the idle state, waiting for CNN to start calculation; S1 state is to write to caches at all levels, including writing input data read from off-chip DRAM into on-chip BRAM, and then reading input data and weight data from BRAM to CPE for calculation; S2 state is the convolution layer calculation of CNN forward reasoning calculation, including convolution calculation, maximum pooling and activation calculation; S3 state is the fully connected layer calculation of CNN forward reasoning calculation, including fully connected calculation and softmax calculation; S4 state is a sign of the end of CNN forward reasoning calculation, indicating that the entire CNN forward reasoning calculation is completed. The specific process of state transition includes: when the cnn_start signal is valid, the accelerator jumps from the S0 state to the S1 state, starts to read the input data from the off-chip DRAM and writes it into the on-chip BRAM, and then reads the input data and weight data from the on-chip BRAM and writes them into the internal cache of the computing unit; when the data writing is completed, the conv_douta_done signal is valid, the accelerator jumps from the S1 state to the S2 state, and starts the first layer of convolution calculation. When the first layer of convolution calculation is completed, the conv_done signal is valid, the accelerator jumps from the S2 state to the S0 idle state, and starts the next layer of convolution calculation; when the two layers of convolution calculation are completed, f When the c_start signal is valid, the accelerator jumps to the S3 state and starts to perform the calculation of the first fully connected layer. When the calculation of the first fully connected layer is completed, the fc_done signal is valid, and the accelerator jumps from the S3 state to the S0 state and starts to perform the calculation of the second fully connected layer. After the calculation of the two fully connected layers is completed, softmax outputs the classification recognition probability of the output results of the second fully connected layer. When the classification recognition probability output by the entire CNN network is written into the on-chip output cache module and read into the off-chip DRAM, the cnn_done signal is valid, and the accelerator jumps from the S3 state to the S4 state, indicating that the entire accelerator has completed the accelerated calculation of CNN forward reasoning.

[0052] like Figure 3 As shown in the figure, the convolution calculation unit is composed of 16 CPEs for parallel calculation. Each CPE includes a convolution module and a pooling activation module, which mainly completes convolution, normalization and pooling activation calculations. The weight data of the two layers of convolution are stored in different weight cache modules respectively. The intermediate cache module includes multiple layers of cache. The parallel calculation results of 16 CPEs in each layer are stored in the intermediate cache modules of 16 layers at the same time. These calculation results are transmitted to the CPE array again in a certain order for the next layer of convolution calculation. This parallel calculation and parallel storage method is used to fully tap the computational parallelism of the neural network and improve the performance of the accelerator.

[0053] like Figure 4As shown, the convolution module is composed of a DSP array, and the size of the DSP array is R×L, where R represents the number of channels for parallel calculation, and L represents the size of the convolution kernel. In this embodiment, the convolution kernel size is 3×3, the number of input channels is 16, and the convolution module adopts a DSP array structure of 4 rows and 9 columns. The 9 DSPs in each row are cascaded to calculate the multiplication operation in one channel, and the 4 rows of DSPs process the multiplication calculation of 4 channels in parallel, and multiplex four times to complete the operation of a convolution kernel. The time difference of 2 clock cycles is set between two adjacent rows of DSPs, and the next channel accumulates the product result of the previous channel to realize the accumulation operation between channels. There is no need to design an additional adder to complete the accumulation between channels, saving the logic resources of FPGA.

[0054] The computation process of the convolution module includes the following steps:

[0055] Step 11, expand the two-dimensional matrix of input data and weight data into a one-dimensional matrix along the rows first and then the columns, and then convert the floating-point data of the one-dimensional matrix into corresponding fixed-point numbers;

[0056] Step 12, input the converted input data and weight data into the DSP array, and perform multiplication operation on the 9 input data and weight data in the convolution window, and then perform accumulation operation among the 4 channels;

[0057] Step 13, add the 9 data accumulated between channels to obtain the calculation result of the first convolution window, and calculate the sliding window backward in sequence according to the convolution step size of 1. After a certain clock cycle, a layer of convolution calculation results is obtained, and it is determined whether the convolution calculation is completed. Otherwise, return to step 12 to continue the calculation until all convolution layers are calculated, and then execute step 14;

[0058] Step 14, normalize the convolution calculation result to obtain the output result of the convolution module, and cache the output result in the intermediate cache module of the cache unit.

[0059] When the CNN network model is different and the number of convolutional layer input channels changes, based on the DSP array structure of this embodiment, while balancing FPGA resource utilization and accelerator computing performance, multiplexing is performed in units of 4 or the number of channels that can be calculated in parallel is expanded by multiples of 4 to complete the calculation.

[0060] The pooling activation module consists of a maximum comparator and a 0-value comparator, such as Figure 5As shown, in this embodiment, the size of the pooling window is 2×2, and the input of the pooling window is the output of the previous convolution module. In this embodiment, the input of the pooling window is exemplified by data x1-x4 to illustrate the calculation process. The input data of the first row of the pooling window is x1 and x2; the input data of the second row is x3 and x4. The comparator makes comparisons in the form of a tree, requiring two levels of comparators. Both data to be compared are 16-bit signed numbers. In this embodiment, the sign bit is judged first. For a positive and a negative signed number, a larger value can be obtained by judging the single bit of the sign bit. Multi-bit comparisons are performed only when the sign numbers are the same. Finally, the activation unit only needs to compare the pooled result with 0 to output the activated result. The calculation process of the pooling activation module includes the following steps:

[0061] Step 21, first perform maximum comparison on the first row x1, x2 and the second row x3, x4 of the pooling window at the same time, and obtain the maximum value y1 of the first row and the maximum value y2 of the second row respectively;

[0062] Step 22: perform maximum comparison on the maximum values ​​y1 and y2 of each row to obtain the maximum pooling result y max ;

[0063] Step 23: The maximum pooling result y max The input data is sent to the 0 value comparator to compare the input data with 0. When the input data is less than 0, the output is 0; when the input data is greater than or equal to 0, the output data keeps the size of the input data unchanged and caches the output data in the intermediate cache module of the cache unit as the input data of the fully connected computing unit.

[0064] like Figure 6 As shown in the figure, the fully connected computing unit consists of a parallel FCPE array and softmax. The calculation results of each layer of full connection are cached in the intermediate cache module. These calculation results are transmitted to the FCPE array again in a certain order for the next layer of full connection calculation. The weight data of the two layers of full connection is stored in two weight cache modules. The FCPE array consists of 8 independent FCPEs to complete the full connection operation. Softmax is used to map the classification results of the last layer of full connection into probability output.

[0065] In this embodiment, the number of input channels of the fully connected layer is 16, and each FCPE is equipped with 16 multipliers, 16 adders, 1 zero-value comparator, and 1 bias register. The 16 multipliers perform multiplication operations simultaneously, and the 16 adders accumulate the multiplication results. In order to improve the throughput of the fully connected calculation, the implementation of the adder adds a pipeline structure, and adopts a five-stage pipeline adder design. Compared with general adders, the addition of a pipeline structure can further increase the operating frequency and improve the calculation speed. Figure 7As shown, each FCPE calculation process includes the following steps:

[0066] Step 31, sending the input data and weight data to 16 multipliers for multiplication operation at the same time to obtain multiplication results P0-P15, where the input data is the output data of the convolution calculation module;

[0067] Step 32, P0~P15 are added in pairs through the first-level adder to obtain the first-level addition results S10~S17, and then added in pairs through the second-level adder to obtain the second-level addition results S20~S23, and then added in pairs through the third-level adder to obtain the third-level addition results S30~S31, and then added in pairs through the fourth-level adder to obtain the fourth-level addition result S40, and then added through the fifth-level adder and the bias register bias to obtain the fifth-level addition result S50. The value of S50 continuously updates the bias register bias value. When all the input data and weight data are multiplied and accumulated, the bias register stops updating and inputs the activation unit to obtain an output result of the fully connected layer. 8 FCPEs are calculated in parallel, reused 16 times to obtain the output of 128 nodes, and all output results are cached in the intermediate cache module of the cache unit as the input data of softmax.

[0068] When the network model is different and the number of input channels of the fully connected layer changes, based on the design of 16 adders and 16 multipliers proposed in this embodiment, while balancing the utilization of FPGA resources and the computing performance of the accelerator, the calculations are completed by multiplexing in units of 16 or expanding in multiples of 16.

[0069] The fully connected computing unit also includes a softmax, which is composed of a maximum comparator, an adder, and a divider. The softmax outputs the classification results of the last fully connected layer in a probabilistic manner. In this embodiment, the output layer is a 6-category softmax. Softmax includes exponential function operations. In hardware implementation, a lookup table is used to convert the exponential function with base e into an exponential function with base 1 / e through formula conversion, and the converted exponential function is divided into a finite number of intervals. For example, 1 / e to the power of 0 is 1, and 1 / e to the power of 6 is only 0.0024. Therefore, the range of the input data can be set to (0, 6), and (0, 6) can be divided into 48 intervals with a segmentation accuracy of 0.001. Each interval is approximated by a fixed coefficient value, and the fixed coefficient value of each interval is stored in the BRAM of the FPGA. For example Figure 8 As shown, the softmax probability output includes the following steps:

[0070] Step 41, use the 6 output data input0-input5 of the last fully connected layer as the input data of softmax, and find the maximum value through the maximum comparator MAX;

[0071] Step 42, calculating the difference between the maximum value and the six input data, reading the fixed coefficient values ​​of the intervals corresponding to the six differences from the BRAM, and accumulating them;

[0072] Step 43, taking the fixed coefficient values ​​of the intervals corresponding to the six differences as numerators and the accumulated results of the coefficient values ​​of the intervals corresponding to the six differences as denominators, and sending them to the divider for calculation to obtain the output results of the six-category probability, and cache the output results in the output cache module of the cache unit.

[0073] In the design of FPGA hardware accelerator, the present invention uses data structure optimization technology to perform 16-bit fixed-point quantization on the floating-point data in CNN. Using a suitable fixed-point bit width for linear quantization can effectively reduce the overhead of storage resources and memory access, improve the performance of the accelerator, and the influence of fixed-point calculation on the final model result can be ignored. Fixed-point numbers are fixed decimal points, and integers are used to represent fixed-point decimals. Since the position of the decimal point is fixed, there is no need to store the position of the decimal point. However, the position of the decimal point needs to be remembered to avoid errors in calculation. This embodiment uses 16-bit binary to represent fixed-point numbers, the highest bit is the sign bit, there are 15 valid bits, there are 0-15 bits after the decimal point, and n bits after the decimal point are called Qn.

[0074] Fig. 9 This is a diagram of the accuracy of the first layer of CNN. The input data of the first layer of convolution is a Q0 format fixed-point number, the weight data is a Q15 format fixed-point number, and the DSP output is set to 48-bit bit width. The result of the product of the input data and the weight data is a Q15 format fixed-point number. Through the software model calculation, all integer bits are retained, and as many decimal places as possible are retained. The product result of the 48-bit Q15 format fixed-point number is truncated to a 16-bit Q1 format fixed-point number. In order to avoid data overflow during the addition process and cause calculation errors, an overflow protection adder design is used for the product result. Because addition does not change the fixed-point format of the data, the output result of the first layer of convolution is a Q1 format fixed-point number.

[0075] The overflow protection design of the subsequent layers is exactly the same as that of the first layer, and the truncation operation is slightly different. The only difference is the number of fixed-point bits of the truncation operation. The specific truncation of each layer is shown in Table 2.

[0076] Table 2 Fixed-point numbers of truncation formats of each layer of CNN network

[0077]

[0078]

[0079] The following is combined with specific experimental results to illustrate that the FPGA accelerator proposed in the present invention has the advantage of high energy efficiency compared to general-purpose processors. Table 3 compares the performance of CNN implemented on FPGA with the results of running on CPU and GPU (where the running data under CPU and GPU are the average values ​​under 10,000 consecutive inputs). The FPGA platform used to implement the proposed accelerator architecture is Xilinx's Zynq UltraScale+ZCU106 development board; in the comparative experiment, the CPU uses Intel(R) Xeon(R) E5-2630 v2 processor with a main frequency of 2.6GHz; the GPU uses NVIDIA GeForce CTX 1080Ti with a core frequency of 2.0GHZ.

[0080] Table 3 Performance comparison of CNN accelerators in FPGA, CPU, and GPU

[0081] platform CPU GPU FPGA Time (ms) 6.79 2.01 0.089 Power consumption (W) 9.78 47.55 3.67

[0082] The accelerator performance evaluation indicators mainly include acceleration ratio, throughput and energy efficiency. Compared with CPU, the FPGA-based accelerator of the present invention can achieve a speed increase of nearly 76 times; compared with GPU, the FPGA-based accelerator can achieve a speed increase of nearly 22 times, but the power consumption is only about 1 / 13 of the GPU power consumption, which has a very high energy consumption advantage. The total number of CNN operations is 8459788, and the clock cycle is 17717. At a clock frequency of 200MHZ, the throughput of the accelerator of the present invention can be obtained to be about 95.5GOPS. The energy efficiency of the accelerator of the present invention reaches 26.02GOPS / W, which has the advantage of high energy efficiency compared with general-purpose processors.

[0083] Table 4 FGPA resource consumption,

[0084] resource use Proportion(%) LUT 126238 / 230400 54.79 FF 119841 / 460800 26.01 BRAM 269 / 312 86.22 URAM 36 / 96 37.50 DSP 578 / 1728 33.45

[0085] Table 4 shows the resource consumption of FGPA. The accelerator design of the present invention fully considers resource reuse, multi-level storage, and pipeline design, effectively reducing the utilization rate of the main resources of FPGA. Because in the process of accelerated calculation, the power consumption of reading data from off-chip DRAM is much greater than the power consumption of reading data from on-chip BRAM, and reading data from on-chip BRAM is beneficial to improving the accelerator frequency and throughput. Therefore, in order to reduce the power consumption overhead of the accelerator and improve the performance of the accelerator, the weight cache is placed entirely on-chip and the input cache is placed off-chip. The problem caused by placing all weight caches on-chip is that the utilization rate of on-chip BRAM resources is high. The present invention reduces BRAM resources as much as possible by reusing cache modules and replacing BRAM with URAM on the ZCU106 development board, and finally controls the BRAM resource utilization rate to about 85%.

Claims

1. A convolutional neural network accelerator based on FPGA, characterized in that: include: A control unit, used to control the state transition and data flow transmission process of the global calculation of the convolutional neural network; Cache unit, used to store the weight data, input data, output data, and calculation results of each layer of convolution and full connection required for calculation; Computational unit, used for parallel computing of convolutional neural networks; The computing unit includes a convolution computing unit and a fully connected computing unit; The convolution calculation unit includes N CPEs for parallel calculation, each CPE includes a convolution module and a pooling activation module; the convolution module is composed of a DSP array, which is used to complete the convolution calculation and normalization operation; the pooling activation module is composed of a maximum comparator and a 0 value comparator, and the pooling activation module is used to perform maximum pooling and activation operations on the output results of the convolution module; The fully connected computing unit includes softmax and M parallel computing FCPEs, each FCPE is composed of a multiplier and an adder, and the FCPE is used to perform multiplication and accumulation operations on input data and weight data; softmax is composed of a divider, an adder and a maximum comparator, and softmax is used to output the classification results of the last fully connected layer in a probabilistic manner; The calculation process of the convolution module includes the following steps: Step 11, expand the two-dimensional matrix of input data and weight data into a one-dimensional matrix along the rows first and then the columns, and then convert the floating-point data of the one-dimensional matrix into corresponding fixed-point numbers; Step 12, input the converted input data and weight data into the DSP array, and perform multiplication operation on the L input data and weight data in the convolution window first, and then perform accumulation operation between R channels; The size of the DSP array is R×L, where R represents the number of channels for parallel computing and L represents the size of the convolution kernel; Step 13, perform an addition operation on the L data accumulated between channels to obtain a convolution calculation result, and determine whether the convolution calculation is completed. If it is completed, execute step 14, otherwise return to step 12 to continue the calculation; Step 14, normalizing the convolution calculation result to obtain the output result of the convolution module, and caching the output result in the intermediate cache module of the cache unit; The calculation process of the pooling activation module includes the following steps: Step 21, performing maximum comparison on the pooling window row by row to obtain the maximum value of each row; Step 22, perform maximum comparison on the maximum value of each row to obtain the maximum pooling result; Step 23, sending the maximum pooling result as input data to the 0 value comparator, comparing the input data with 0, when the input data is less than 0, the output is 0; when the input data is greater than or equal to 0, the output data keeps the size of the input data unchanged, and caches the output data in the intermediate cache module of the cache unit; The adder of FCPE adopts a five-stage pipeline adder. Each FCPE calculation process includes the following steps: Step 31, sending the input data and weight data to each multiplier for simultaneous multiplication to obtain each multiplication result; Step 32, using a five-stage pipeline adder to accumulate all multiplication results and bias registers, and using the addition results to continuously update the value of the bias register. When all input data and weight data are multiplied and accumulated, the bias register stops updating, inputs the activation unit, obtains the output result of the fully connected layer, and caches the output result of the current layer in the intermediate cache module of the cache unit; Softmax includes the operation of exponential function. In hardware implementation, a lookup table is used to convert the exponential function with base e into an exponential function with base 1 / e through formula conversion. The converted exponential function is divided into a finite number of intervals. Each interval is approximated by a fixed coefficient value. The fixed coefficient value of each interval is stored in the BRAM of the FPGA. The output process of softmax includes the following steps: Step 41, using the output data of the last fully connected layer as the input data of softmax, and finding the maximum value through the maximum comparator; Step 42, calculating the difference between the maximum value and the input data, reading the fixed coefficient value of the interval corresponding to the difference from the BRAM, and accumulating; Step 43, taking the fixed coefficient value of the interval corresponding to the difference as the numerator and the accumulated result as the denominator, and sending them to the divider for calculation to obtain the probability output result, and cache the output result in the output cache module of the cache unit.

2. The convolutional neural network accelerator according to claim 1, characterized in that: The cache unit includes an off-chip DRAM, an input cache module, an output cache module, a weight cache module and an intermediate cache module. The off-chip DRAM is used to store off-chip data, the input cache module is used to store data read from the off-chip DRAM, the output cache module is used to store the output data of the fully connected calculation module, the weight cache module is used to store weight parameters, and the intermediate cache module is used to store the calculation results of each layer of convolution and each layer of full connection.

Images