Convolutional neural network acceleration method based on FPGA configuration and easy to improve parallelism

Abstract

The application relates to a convolutional neural network acceleration method based on FPGA configuration and easy promotion of parallel degree, and belongs to the field of deep learning, and comprises the following steps: S1, data is rearranged at an ARM end and stored into external storage DDR; S2, after a start calculation instruction issued by the ARM is accepted by FPGA, the FPGA reads data in the DDR through burst transmission and places the data into an on-chip buffer; S3, when data in the buffer is ready, a convolution calculation module in the FPGA starts to acquire data and performs calculation; and S4, a STORE module of the FPGA is activated, quantized and returns output data to the DDR for next convolution.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of deep learning, and relates to a convolutional neural network acceleration method based on FPGA, which is configurable and easy to improve parallelism. BACKGROUND

[0002] A convolutional neural network (CNN) is a kind of feedforward neural network containing convolution calculation and having a deep structure, and is one of important components of deep learning. The convolutional neural network has certain representation learning ability and can improve the ability of extracting image features by increasing the depth of the model. In recent years, the convolutional neural network has achieved great results in image analysis, automatic driving, natural language processing and speech recognition, and has become a research hotspot in all circles. Initially, the inference stage of the convolutional neural network is usually implemented on software, and with the continuous replacement of various network models, the operation speed and recognition accuracy are higher and higher, but the inference time using CPU is doubled with the increase of the depth of the network; and due to the characteristics of large size and large number of parameters of the convolutional neural network, it is difficult to be directly deployed on an edge computing platform.

[0003] At present, in the aspect of processing deep learning AI algorithms, the industry mainly uses general-purpose chips such as GPU and FPGA suitable for parallel computing to realize acceleration. At the same time, some chip companies have begun to design ASIC special-purpose chips for AI algorithms, such as Google TPU and Horizon BPU. Before the chip products are mass-produced and put into batch production, using the existing general-purpose chips such as GPU and FPGA can avoid the high investment and high risk of specially developing custom chips (ASIC). Due to the special structure of the convolutional neural network, a large number of multiplication and addition operations are involved, and a large number of memory data accesses are also involved, and the operation speed of the traditional general-purpose processor in the CNN inference stage is not ideal, and the GPU also has the disadvantages of high power consumption and high price. Therefore, deploying AI algorithms using FPGA has certain advantages.

[0004] CNN models implemented in FPGAs primarily handle the convolution operations, which involve numerous multiplication and addition operations. Other parts are handled by the ARM processors in the SoC, allowing each to leverage its strengths and freeing up FPGA resources for convolution. Currently, convolution calculations deployed in FPGAs are mostly performed sequentially using a sliding window approach. However, this method consumes significant time in transmitting input, weights, and offsets, and it doesn't adapt well to changes in kernel size. Furthermore, forcibly increasing data transmission bandwidth under this approach makes the control logic extremely complex. Due to bandwidth limitations, FPGA DSP resources are not fully utilized, wasting resources without improving parallelism and limiting convolution calculation speed. Therefore, using the sliding window convolution method to improve CNN model performance may face challenges such as poor adaptability, complex control logic, and slow data transmission, making it difficult to increase convolution speed. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a configurable FPGA-based method for accelerating convolutional neural networks with easily improved parallelism. It not only designs a new CNN convolution calculation method on the FPGA side, but also rearranges the data on the ARM side according to the FPGA calculation method, thereby simplifying its control logic and making full use of the high bit width advantage of the Avalon bus, while also improving the system parallelism more easily.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] An FPGA-based method for accelerating convolutional neural networks with easily improved parallelism includes the following steps:

[0008] S1: The data is rearranged on the ARM side and stored in external DDR memory;

[0009] S2: After receiving the start calculation instruction from the ARM, the FPGA reads the data in the DDR through burst transfer and puts it into the on-chip buffer;

[0010] S3: Once the data in the buffer is ready, the convolution calculation module in the FPGA begins to acquire the data and perform calculations;

[0011] S4: Activate, quantize, and return the output data to DDR using the STORE module of the FPGA for the next convolution.

[0012] Further, the data is rearranged at the ARM end according to the convolution calculation mode deployed at the FPGA end, and the input feature map data and the weight data are quantized to 8 bits.

[0013] For the input feature map data, the input channels are cut into in_c / p slices, and after one slice is completely transmitted, the next slice is transmitted, wherein in_c represents the number of all input channels, and p represents the number of input channels in one input channel slice, and p is also the number of data transmitted in a single cycle in the high-bit-width transmission mode.

[0014] For the weight data, the number of convolution kernels is the product of the number of input slice channels and the number of output slice channels, the weight corresponding to the first pixel in the first channel in the input slice is output in sequence, the input channel number is traversed, and finally the kernel size is traversed kernel_size times, so that all weights of the corresponding slice are output.

[0015] Further, in the step S2, the FPGA inputs the feature map and the weight data through the LOAD module, the LOAD module is composed of a selector and a plurality of buffers, and plays a role in cross-clock domain processing and avoids the waiting time of data transmission through the PingPong Buffer, the PingPong Buffer uses two buffers in a transmission module to alternately read and write data, so that the data is transmitted without interruption.

[0016] Further, the bit width of all input feature map data and weight data in the LOAD module is set to 128 bits to match the data bit width of the Avalon bus, the bit width of the weight data output from the LOAD module to the convolution calculation module is 16 x the number of channels in a single input channel slice after sparsification x the number of channels in a single output channel slice bit, which corresponds to all weights in the corresponding input channel and output channel slice; the weight data and the input feature map data output from the LOAD module to the convolution calculation module correspond to each other.

[0017] Further, the calculation mode of the convolution calculation module in step S3 is to match the input channel slice with the data bit width of the Avalon bus, so that all the data of the same position of all input channels in a slice are taken out in one period, and at the same time, all the weights corresponding to the input channel slice and the output channel slice are taken out in this period, so as to calculate a part of the multiplication and addition result of the corresponding output value in the output channel slice; then the weight data is kept unchanged, the input feature map data is changed, and a part of the value of all data in the slice output channel is obtained after the traversal is completed; then the same operation is performed after the weight data is changed, and the complete value in the corresponding output channel slice is obtained after the weight data is changed repeatedly kernel_size times.

[0018] Further, the weight data and the input data are int8 types, and 32 bits are used to accommodate the accumulated result, and the STORE module quantizes the 32-bit output data to 8-bit data and then stores it in the DDR for use by the next convolution layer.

[0019] The beneficial effects of the present application are that the technical solution not only designs a new CNN convolution calculation mode on the FPGA end, but also rearranges the data on the ARM end according to the FPGA calculation mode, so as to simplify the control logic and make full use of the high bit width advantage of the Avalon bus, and at the same time, the system parallelism can be improved more simply. The CNN accelerator designed by using the convolution calculation mode and data rearrangement of the present application can match the convolution layer under different convolution kernel sizes, input and output channel sizes and different step lengths, and only needs to send relevant configuration commands from the ARM to the FPGA, which can well solve the shortcomings of the traditional CNN accelerator, such as low adaptability, low parallelism, complex control logic and slow acceleration effect. The CNN accelerator designed by using the convolution calculation mode and data rearrangement can match the convolution layer under different convolution kernel sizes, input and output channel sizes and different step lengths, and only needs to send relevant configuration commands from the ARM to the FPGA.

[0020] Other advantages, objects, and features of the present application will be understood in view of the following specification, and will be apparent to those skilled in the art from the teachings of the following specification and the accompanying drawings. The objects and other advantages of the present application will be realized and attained by the embodiments particularly pointed out in the specification. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, in which:

[0022] Figure 1 is a system overall block diagram;

[0023] Figure 2 The flow chart for rearranging the input feature map data;

[0024] Figure 3 The flow chart for rearranging the weight data;

[0025] Figure 4 The block diagram of the LOAD module related to the feature map data;

[0026] Figure 5 The block diagram of the LOAD module related to the weight data;

[0027] Figure 6 The schematic diagram of the convolution calculation mode in the channel dimension;

[0028] Figure 7 The flow chart of the overall data processing;

[0029] Figure 8 The principle diagram of the PingPong Buffer;

[0030] Figure 9 The resource report table;

[0031] Figure 10 The information printed on the ARM side;

[0032] Figure 11 The picture recognition effect verified on the Intel Cyclone V 5CSEBA6U23I7. DETAILED DESCRIPTION

[0033] The present application is further explained in the following examples, which should not be construed as limiting the scope of the application. The following examples provide what are presently believed to be the best modes of practicing the application. The examples do not, however, exhaustively illustrate the application in all of its potentially useful embodiments. Other embodiments of the application will be apparent to those skilled in the art in view of this disclosure.

[0034] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0035] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it is understood that if the orientations or positional relationships indicated by the terms "upper", "lower", "left", "right", "front", "back" and the like are based on the orientations or positional relationships shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only used for exemplary illustration, and cannot be understood as a limitation on the present application, for those skilled in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0036] The present application provides a configurable, easy-to-improve parallelism-based convolutional neural network acceleration method based on FPGA, which solves the following three problems existing in the prior art: the traditional CNN accelerator is not strong in adaptability to different convolution kernels and different input and output channels; for the input feature map data without rearrangement, the control logic of the traditional CNN accelerator is complex and time-consuming when the parallelism is high; the calculation parallelism in the traditional CNN accelerator is limited by the size of the convolution kernel, and if the parallelism is forcibly improved, the control logic will also be very complex.

[0037] The present application mainly designs a new CNN network convolution calculation method based on FPGA, in order to cooperate with this calculation method, the data needs to be rearranged in advance at the ARM end, so that when the FPGA reads the data in the DDR, it can fully utilize the bus bandwidth, break the bottleneck of sliding window data bandwidth, and also improve the parallel execution efficiency of the CNN accelerator and simplify the control logic by comprehensively using optimization techniques such as operator fusion, double buffering and pipeline.

[0038] First, from the overall point of view, as shown in the accompanying Figure 1 As the data volume of input feature maps and weight offsets and other data in the convolutional neural network is generally large, and the on-chip storage of FPGA is very precious and insufficient to hold all the data, the design uses external DDR storage to store data. Before writing data into DDR, data rearrangement needs to be performed at the ARM end according to the convolution calculation method deployed at the FPGA end, and the feature map input and weight are quantized to 8 bits to reduce the data transmission volume and calculation volume of FPGA.

[0039] Taking rearrangement of input feature map data as an example, the application does not transmit all data of one input channel in the order from left to right and from top to bottom as in the traditional transmission sequence, and then transmit data of the next channel. Instead, the input channel is cut into in_c / p slices, and after transmitting one slice completely, the next slice is transmitted. Wherein in_c represents the number of all input channels, and p represents the number of input channels in one input channel slice, and p is also the number of data transmitted in a single cycle in the high-bit-width transmission mode. For example, when a 128-bit transmission port is used, the actual bit width of the data is 8 bits, and then P = 16, which means that P input channel data is transmitted in each cycle. Note that when the input channel cannot be divided by P, the integer part is taken and 0 values are filled. When transmitting one slice, the application first transmits data of the same position of all channels in sequence according to the input channel as the dimension, and then transmits data of all channels in the next position, as shown in FIG. 8. Figure 2 The weight rearrangement mode is similar to the input rearrangement mode. Taking the number of input slice channels * the number of output slice channels of the convolution kernel as a unit, the weights corresponding to the first pixel in the first channel in the input slice are output in sequence first with the output slice channel as the dimension, then the input channel number is traversed, and finally the kernel_size is traversed, so that all weights of the corresponding slice are output. Through this data rearrangement mode, the present application eliminates the relatively complex control logic required when importing on-chip storage, and can well cooperate with the newly proposed convolution calculation mode to speed up the operation and simplify the logic control. Similarly, the weight data also adopts a similar rearrangement mode to cooperate with the rearrangement mode of the input feature map, so that the data reading and calculation of the CONV module in the FPGA are more convenient, as shown in FIG. 9. Figure 3

[0040] On the FPGA, the design mainly includes three parts:

[0041] LOAD module: The module performs input of feature map and weight data, and is mainly composed of a selector and a plurality of buffers, as shown in FIG. 10. Figure 4 5 The module plays a role in cross-clock domain processing and avoids data transmission waiting time through the PingPong_Buffer mode; and after the data is cached to the buffer, the Conv module can flexibly and quickly take out the required data for calculation according to the required sequence (related to the configured step length).

[0042] ​​The bit width of all DATA and WEIGHT in the module is set to 128 bits to match the data bit width of the Avalon bus, thereby improving transmission efficiency and parallelism of the calculation. However, it should be noted that the data bit width of weight_out is 16*Dense_input_channel_tile*Output_channel_tile bits, which corresponds to all weights in the corresponding input channel, output channel tile; it should be noted that the data output by weight_out and Data_out correspond to each other, so that the correct convolution operation can be performed.

[0043] The CONV module: the traditional sliding window convolution method cannot well support the requirement of configurable convolution kernel size, and the complex control logic of the method will change greatly with the change of the convolution kernel size, making it more difficult to meet the requirement of configurable convolution kernel. Assuming that the Avalon bus data bit width is 128 bits, the sliding window method is as follows: in the first Avalon data reading period, the first 9 convolution input data correspond to kernel-1 weight parameters, and the last 7 input data correspond to kernel-2 weight parameters; in the second Avalon data reading period, the first 2 convolution input data correspond to kernel-2 weight parameters, the middle 9 input data correspond to kernel-3 weight parameters, and the last 5 input data correspond to kernel-4 weight parameters. Thus, it can be seen that the control logic is complicated.

[0044] The convolution calculation method adopted by the application is to match the input channel slice with the data bit width of the Avalon bus, so that all the data of the same position in all input channels in a slice are taken out in one period, and at the same time, all the weight values corresponding to the input channel slice and the output channel slice are taken out in the period, so as to calculate a part of the multiplication and addition result of the corresponding output value in the output channel slice, as shown in the accompanying Figure 6 After that, the weight data remains unchanged, the input feature map data is changed, and after the end of this iteration, a part of the data in the slice output channel is obtained; after the weight data is changed, the same operation is performed, and after the weight data kernel_size(convolution kernel size) times are changed, all the complete values in the corresponding output channel slice are obtained.

[0045] This way does not transmit data in a transmission cycle with the dimension of the convolution kernel size, but with the dimension of the number of channels in an input channel slice and data zero padding is performed, so this design can well adapt to various sizes of kernel_size, and even when the input and output channel number, convolution kernel step and input feature map size change, it can also be well adapted, and does not require overly complex control logic. And the parallelism of the accelerator is no longer directly related to the size of the convolution kernel, so the parallelism can be set arbitrarily without complex control logic.

[0046] The STORE module mainly activates, quantizes and returns the output data to the DDR for the next convolution. Because only a small amount of hardware resources is needed to complete the multiplication and addition of low-bit-width data, the weight data and input data used in this design are of int8 type; considering that the product of two 8-bit data is 16-bit, and the addition of multiple 16-bit numbers, this design uses 32-bit to accommodate the accumulated results to prevent numerical overflow. Since the output results of the previous convolution layer need to be used as the input feature map data of the next convolution layer, the 32-bit output data needs to be quantized to 8-bit data before being stored in the DDR for use by the next convolution layer.

[0047] In many CNN accelerators deployed on FPGAs in the past, data quantization is processed on the ARM side, but this way will make ARM read and write DDR frequently, which will waste a lot of time in data interaction when there are many convolution layers. Therefore, in order to further improve the overall system operation speed, this design moves the data quantization to the FPGA for design, as shown in the accompanying Figure 7

[0048] In the overall system, this design uses a pipelining way to make the LOAD, CONV and STORE modules pipelined, reducing the system waiting time. And when it comes to data interaction, this design uses PingPong Buffer to improve data transmission efficiency, and tries to keep the whole system in working condition, so as to realize the pipelining of data access and convolution calculation. The working principle of PingPong Buffer is shown in the accompanying Figure 8

[0049] ​​After the data is stored in the DDR and the FPGA receives the start calculation instruction issued by the ARM, the FPGA starts to read the data in the DDR by burst transmission and put it into the on-chip Buffer (the data bit width of the Buffer can be changed according to the required parallelism and slicing method, thereby providing conditions for improving parallelism), when the data in the Buffer is ready, the convolution calculation module in the FPGA can start to take data and perform calculation, and due to the design of PingPong_Buffer, the data load and convolution calculation in the present design can be performed simultaneously.

[0050] Since the data bit width set in the Buffer is high, when the FPGA starts convolution, the input feature map data of the input channel in a slice can be taken out in one period, and the corresponding input slice channel output slice channel weights can be taken out at the same time. When performing calculation, we first keep the weight data unchanged, traverse all the corresponding values of the input channel in the slice, then change to the next weight data to continue traversal, and accumulate the corresponding data in the front and back traversal processes.

[0051] Since the input feature map and the weight are both int8 data types, and multiple multiplication and addition operations are required, in order to prevent data overflow, the data type of the accumulated result is defined as int32. But since the result of this convolution layer needs to be used as the input of the next convolution layer, before storing the accumulated result in the DDR, the 8-bit DoReFa quantization algorithm is used to quantize int32 to int8, and thus a complete convolution operation of one layer is completed.

[0052] In actual application cases, the acceleration method described in the present application is verified on an Intel Cyclone V 5CSEBA6U23I7, and in the case of various hardware resources, it only takes 59 ms to calculate all the convolution layers in the MobileNet V1 SSD model, and there is no calculation error. And after deploying the Paddle_lite framework of Baidu on the ARM end in the SOC system, image recognition and classification can be realized, as shown in the accompanying Figures 9-11 .

[0053] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, and all should be covered in the scope of the claims of the present application.

Claims

1. A method for accelerating a convolutional neural network based on FPGA, which is configurable and easy to upgrade parallelism, characterized in that: The method comprises the following steps: S1: rearranging data at the ARM end and storing in external storage DDR; the step S1 of rearranging data at the ARM end specifically comprises rearranging data at the ARM end according to the convolution calculation mode deployed at the FPGA end, and quantifying both the input feature map data and the weight data to 8 bits; For the input feature map data, the input channels are cut into in_c / p slices, and after a slice is completely transmitted, the next slice is transmitted, wherein in_c represents the total number of input channels, and p represents the number of input channels in a single input channel slice, and p is also the number of data transmitted in a single cycle in the high-bit-width transmission mode; When the input channels cannot be evenly divided by P, the result is rounded up and filled with 0 values; For the weight data, the number of convolution kernels is the product of the number of input slice channels and the number of output slice channels, the weight corresponding to the first pixel in the first channel in the input slice is output in sequence, then the input channel number is traversed, and finally the kernel size kernel_size is traversed, so that all weights of the corresponding slice are output S2: after the FPGA receives the start computing instruction sent by the ARM, the FPGA reads the data in the DDR through burst transmission and puts the data into the on-chip Buffer; S3: when the data in the Buffer is ready, the convolution calculation module in the FPGA starts to obtain data and perform calculation; S4: the STORE module of the FPGA is activated, quantized, and the output data is returned to the DDR for the next convolution.

2. The FPGA-based configurable and easy-to-promote parallelism convolutional neural network acceleration method according to claim 1, characterized in that: In the step S2, the LOAD module is used to input the feature map and the weight data, the LOAD module is composed of a selector and a plurality of Buffers, and plays a role in cross-clock domain processing and avoids the waiting time of data transmission through the PingPong Buffer, the PingPong Buffer uses two Buffers in a transmission module to alternately read and write data, so that the data is transmitted without interruption.

3. The FPGA-based convolutional neural network acceleration method with configurable, easy-to-promote parallelism according to claim 2, characterized in that: The bit width of all input feature map data and weight data in the LOAD module is set to 128 bits to match the data bit width of the Avalon bus, the bit width of the weight data output from the LOAD module to the convolution calculation module is 16 x the number of channels in a single input channel slice after sparsification x the number of channels in a single output channel slice, which corresponds to all weights in the corresponding input channel and output channel slice; The weight data output from the LOAD module to the convolution calculation module and the input feature map data correspond to each other.

4. The FPGA-based convolutional neural network acceleration method with configurable, easy-to-promote parallelism according to claim 1, characterized in that: The calculation mode of the convolution calculation module in step S3 is to match the input channel slice with the data bit width of the Avalon bus, so that all data in the same position of a slice are taken out in a cycle, and all weights corresponding to the input channel slice and the output channel slice are taken out in the cycle, so as to calculate a part of the multiplication and addition result of the corresponding output value in the output channel slice. After that, keep the weight data unchanged, change the input feature map data, and traverse the end to get a part of the value of all data in the slice output channel; Then change the weight data and perform the same operation, repeat the change of weight data kernel_size times to get all complete values in the corresponding output channel slice.

5. The FPGA-based configurable, easy-to-promote parallelism-based convolutional neural network acceleration method according to claim 1, characterized in that: The weight data and input data are of int8 type, and 32 bits are used to accommodate the accumulated results. The STORE module quantizes the 32-bit output data to 8-bit data and then stores it in DDR for use by the next convolution layer.

Images