FPGA-based convolutional neural network acceleration circuit design

Abstract

The invention discloses a convolutional neural network accelerator based on an FPGA, and belongs to the field of deep learning acceleration. Comprising an FPGA-based convolutional layer acceleration circuit design, a pooling layer acceleration circuit design, a full connection layer acceleration circuit design and an FPGA-based zero padding implementation mode. The convolution layer acceleration circuit, the pooling layer acceleration circuit and the full connection layer acceleration circuit are mainly used for convolution calculation, pooling calculation and full connection calculation respectively. And according to the structure of the convolutional neural network, the acceleration circuits of the modules are cascaded. According to the method, the convolutional neural network can be accelerated with relatively low power consumption, so that the energy efficiency ratio and the acceleration performance of the special neural network accelerator are improved.

Description

Technical Field

[0001] The present invention belongs to the field of convolutional neural network acceleration, and specifically relates to a convolutional neural network accelerator based on FPGA. Background Art

[0002] With the rapid development of the Internet and the advent of the big data era, artificial intelligence has been widely used and developed. Deep learning has become the core technology and main research and development direction for realizing artificial intelligence, and is widely used in big data analysis, changing traditional machine learning methods, image recognition and vision, and other fields. At present, with the continuous development of convolutional neural network models, problems such as long operation time and high operation complexity of convolutional neural networks have gradually been exposed. FPGA has become a strong competitor of GPU in the field of accelerating deep learning with its advantages of parallel computing, small size and low power consumption. However, a large number of FPGA-based acceleration circuit designs are currently mainly aimed at accelerating the convolution layer, and the circuit design for the pooling layer and the fully connected layer is still imperfect. Therefore, this application proposes a complete set of circuit designs for convolutional layers, pooling layers and fully connected layers based on FPGA, and also provides an FPGA-based Padding implementation method to improve the energy efficiency and acceleration performance of dedicated accelerators. Summary of the invention

[0003] This application proposes an FPGA-based convolutional layer, pooling layer, and fully connected layer acceleration design method, as well as an FPGA-based Padding implementation method. According to the structure of the convolutional neural network, the circuits of each module are cascaded to improve the computing performance of the accelerator.

[0004] The present invention proposes a convolution layer acceleration circuit structure design based on FPGA, including: a delay device, a multiplier, a counter and an on-chip storage BRAM. The delay device is used to delay the data flow; The multiplier is used to multiply each feature map data stream by the corresponding convolution kernel weight parameter; The adder is used to add the results of the data streams of each branch after passing through the multiplier; The counter is used to generate an enable signal flow; The on-chip storage BRAM is used to store convolution kernel weight parameters and store calculated convolution result values.

[0005] The present invention proposes a pooling layer acceleration circuit structure design based on FPGA, including: a delayer, a comparator, a data selector, a counter and an on-chip storage BRAM. The delayer is used to delay the data flow; The comparator is used to compare the size of the parameters of each branch data stream; The data selector is used to select a certain data path as output at the current moment; The counter is used to generate an enable signal flow; The on-chip storage BRAM is used to store the pooled result value.

[0006] The present invention proposes a fully connected layer acceleration circuit structure design based on FPGA, including: a multiplier, an adder, a counter, a data selector and an on-chip storage BRAM. The multiplier is used to multiply the input data stream by the corresponding fully connected matrix data stream parameter; The adder is used to accumulate the multiplied data stream; The data selector is used to implement a reset function; The on-chip storage BRAM is used for fully connected layer matrix weight parameters and storing calculated result values; The counter is used to generate an enable signal flow.

[0007] The present invention proposes a Padding implementation method based on FPGA, including: on-chip storage BRAM, which is used to write and read data stream elements that need to perform Padding operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a structural diagram of a convolutional layer acceleration circuit based on FPGA provided in an embodiment of the present application; Figure 2 This is an example diagram of the implementation of the convolution acceleration process provided by this application; Figure 3 This is a diagram of the FPGA-based pooling layer acceleration circuit structure provided by this application; Figure 4 This is an example diagram of the implementation of the pooling acceleration process provided by this application; Figure 5 This is a diagram of the fully connected layer acceleration circuit structure based on FPGA provided by this application; Figure 6 This is an example diagram of the implementation of the full connection acceleration process provided by this application; Figure 7 This is an example diagram of the implementation process of Padding based on FPGA provided by this application; Figure 8 This is an example diagram of the implementation of the operation process between modules on the chip provided by this application. DETAILED DESCRIPTION

[0009] In order to make the technical process and specific implementation method of this application clearer, the following content will provide a detailed description and explanation of the implementation method of this application in conjunction with the accompanying drawings.

[0010] The structure diagram of the convolution layer acceleration circuit based on FPGA is as follows Figure 1The convolution module consists of multiple delays, adders, and multipliers. The module uses delays to multiplex data in parallel to improve computational efficiency and performance. The total number of delays depends on the number of elements in a convolution kernel, which is determined by: = (number of rows of the convolution kernel matrix - 1) × number of columns of the convolution kernel matrix, delay = (the number of columns of the convolution kernel matrix - 1). The delay time is equal to the number of columns of the input feature map matrix. The delay time is a fixed value of 1. The convolution acceleration process implementation example provided in the embodiment of the present application is shown in FIG. Figure 2 Assume that the matrix size of the input feature map is 4×4 and the convolution kernel size is 3×3. In this case, the delay The delay time is 4, the quantity is 6, and the delay The delay time is 1 and the quantity is 2.

[0011] The convolution acceleration process implementation example provided in the embodiment of the present application is shown in FIG. Figure 2 As shown. First, the input feature map matrix elements are read sequentially into an input data pipeline to obtain the first data stream. This data stream is multiplexed into multiple data streams through different delay devices. Multiple data streams will be multiplied by the corresponding elements of the convolution kernel respectively, and then the multiplication results are added to obtain a convolution operation value. It should be noted that not every convolution value calculated is valid. Therefore, an enable signal pipeline is introduced in this design to determine whether the result of the convolution calculation is valid at the current moment. When the enable signal is high, it means that the current result is a valid value, otherwise it is invalid.

[0012] The structure diagram of the pooling layer acceleration circuit based on FPGA is as follows Figure 3 As shown in Figure 2 (taking the maximum pooling method as an example). The design principle of the pooling layer acceleration circuit is similar to that of the convolution layer acceleration circuit. It is also composed of delays to form different branches to obtain multiple data pipelines. The total number of delays depends on the size of the pooling filter window, which is determined by: = (number of rows of pooling filter window - 1) × number of columns of pooling filter window, delay = (number of columns of the pooling filter window - 1). The delay time is equal to the number of columns of the input feature map matrix. The delay time is fixed to 1.

[0013] The example diagram of the implementation of the pooling operation acceleration process provided in the embodiment of the present application is as follows Figure 4As shown. When the input data stream enters the pooling module, multiple data streams are obtained under the action of each delay. These data streams will then pass through two branches. One branch is through the data selector and the other branch is through the comparator. The comparator will output according to the current input and control the data selector to select the maximum value as the output at the current moment. This design principle can also be used for other pooling methods such as average pooling. Similarly, an enable signal pipeline is introduced in this design to determine whether the result of the pooling calculation is valid at the current moment. When the enable signal is high, it means that the current result is a valid value, otherwise it is invalid.

[0014] This application proposes a specific FPGA-based fully connected layer circuit structure, such as Figure 5 As shown, the circuit mainly consists of a multiplier, an adder, a data selector and a counter.

[0015] The following is an example of the implementation of the full connection acceleration process provided in the embodiment of the present application: Figure 6 As shown, the input data stream is multiplied by the weight data stream, and the result is transmitted to the adder. The output of the adder is divided into two channels, one of which is used as the output result, and the other is connected to the data selector. The data selector will judge whether to output 0 or select the output from the adder at the current moment according to the enable signal flow. The output of the data selector is connected to the input of the adder to form a feedback loop. In this module, this design introduces an enable signal, and its function is as follows: It is used to determine whether the current output is valid. The high level of the enable signal is used to indicate that the current output is valid, otherwise it is invalid. When the rising edge of the enable signal arrives, the data selector outputs 0 to complete the reset function.

[0016] The design of the enable signal pipeline in the present application is generated by a counter, and it is necessary to calculate and infer the high level and low level moments of the enable signal in advance.

[0017] The implementation example of the Padding process based on FPGA provided in the embodiment of the present application is shown in FIG. Figure 7 As shown. In order to ensure that the size of the input feature map does not change after the convolution operation and to effectively extract edge information, it is usually necessary to add zeros around the input feature map. The FPGA-based Padding implementation proposed in this application is implemented based on the BRAM IP core. In the BRAM IP core, if an address that has not been written to data is accessed, the data read at this address is 0. Based on this feature, pipeline data can be written to a specific address of the BRAM IP core, and then the addresses are read sequentially to obtain the data stream after Padding. This process is shown in Figure 7As shown. Assume that the matrix data size of the original feature map is 2×2, and the matrix data size after padding is 4×4. First, convert the original feature map from matrix form to data pipeline form, and then write each element in the data stream to the specific address of the BRAM IP core, and the address values ​​are 5, 6, 9, and 10 respectively. Finally, read the data from address 0 to 15 in sequence, which forms the data stream after padding.

[0018] The circuit structure composition and process examples of the present invention are explained in detail above. It should be noted that the present invention is not limited to the specific examples shown above, and any changes and modifications made by any technician familiar with the field without departing from the technical solution of the present invention, or modified into equivalent examples with equivalent changes, do not affect the essential content of the present invention; therefore, any simple modification, equivalent change and modification made to the above examples based on the technical essence of the present invention that departs from the content of the technical solution of the present invention, shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A convolutional layer acceleration circuit based on FPGA, characterized in that: Includes counters, delays, multipliers, adders and on-chip storage BRAM; Delay device, used to delay data flow; A multiplier, used to multiply each data stream by the corresponding convolution kernel weight parameter; An adder, used for adding the results of multiplying the data stream of each branch and the convolution kernel weight parameter; A counter for generating an enable signal flow; On-chip storage BRAM is used to store convolution kernel weight parameters and store calculated convolution result values.

2. The convolution layer acceleration circuit according to claim 1, characterized in that: Contains two types of delay time delay, the total number of delay = (the number of elements in the convolution kernel - 1), where the delay The number of delays = (the number of rows of the convolution kernel matrix - 1) × the number of columns of the convolution kernel matrix, =(the number of columns of the convolution kernel matrix - 1), and the delay The delay time is equal to the number of columns of the input feature map matrix. The delay time is fixed to 1.

3. The convolution layer acceleration circuit according to claim 1, characterized in that: The design of the enable signal pipeline is generated by a counter, and it is necessary to calculate and infer the high and low times of the enable signal in advance. When the enable signal is high, it means that the output result value at the current moment is valid. At this time, the data will be written into BRAM for storage, otherwise it is considered an invalid value.

4. A pooling layer acceleration circuit based on FPGA, characterized in that: Includes delay, comparator, data selector, counter and on-chip storage BRAM; Delay device, used to delay data flow; A comparator, used for comparing the sizes of data streams of each branch; Data selector, used to select a certain data as output at the current moment; A counter for generating an enable signal flow; On-chip storage BRAM is used to store the pooled result values.

5. The pooling layer acceleration circuit according to claim 4, characterized in that: Contains two delays with different delay times. The total number of delays depends on the size of the pooling filter window, which is determined by: The number of = (the number of rows of the pooling filter window - 1) × the number of columns of the pooling filter window, the delay = (number of columns of the pooling filter window - 1), where the delay The delay time is equal to the number of columns of the input feature map matrix. The delay time is fixed to 1.

6. The pooling layer acceleration circuit according to claim 4, characterized in that: The design of the enable signal pipeline is generated by a counter, and it is necessary to calculate and infer the high and low times of the enable signal in advance. When the enable signal is high, it means that the output result value at the current moment is valid. At this time, the data will be written into BRAM for storage, otherwise it is an invalid value.

7. The pooling layer acceleration circuit according to claim 4, characterized in that: The pooling method includes but is not limited to maximum pooling, and the comparator and data selector included therein can be replaced by other computing devices to perform other pooling operations.

8. A fully connected layer acceleration circuit based on FPGA, characterized in that: Contains multipliers, adders, data selectors, counters, and on-chip storage BRAM; A multiplier, used for performing a multiplication operation on an input data stream and a corresponding weight parameter data stream; An adder, used to accumulate the multiplied data stream; A counter for generating an enable signal flow; Data selector, used to realize reset function; On-chip storage BRAM is used to store the fully connected layer matrix weight parameters and store the calculated result values.

9. The fully connected layer acceleration circuit according to claim 8, characterized in that: The design of the enable signal pipeline is generated by the counter, and it is necessary to calculate and infer the high and low level moments of the enable signal in advance. On the one hand, when the enable signal is high, it means that the output result value at the current moment is valid. At this time, the data will be written into the BRAM for storage, otherwise it is an invalid value; on the other hand, when the rising edge of the enable signal arrives, the data selector outputs 0 to complete the reset function.

10. A Padding implementation method based on FPGA, characterized in that: It contains the following steps: (1) Write the data that needs to be padded to the specified address in the on-chip storage BRAM; (2) Read the addresses of the on-chip BRAM storage in sequence. The number of read addresses depends on the number of data elements after Padding.

Images