A two-dimensional multi-channel convolution hardware accelerator based on FPGA

Abstract

The present invention relates to a two-dimensional multi-channel convolution hardware accelerator based on FPGA, comprising: a control unit, a bias cache, a weight cache, an input feature cache, a convolution cache, a weight pre-read register group, a PE array, a nonlinear unit, a second gate and a third gate; the feature cache is connected to the PE array; the weight cache is connected to the PE array through the weight pre-read register group; the bias cache and the convolution cache are connected to the PE array through the third gate, and the output end of the PE array is connected to the convolution cache and the nonlinear unit through the second gate; the input feature cache, the bias cache and the weight cache are loaded with data; the weight pre-read register group pre-reads and stores the weight cache; the PE array writes the input feature, the pre-read stored weight data, the bias data or the convolution intermediate result to perform convolution operation, writes the convolution intermediate result to the convolution cache, and outputs the convolution final result after the nonlinear unit is activated. The present invention realizes efficient calculation of any scale convolution layer in CNN.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning acceleration circuits, and specifically relates to a two-dimensional multi-channel convolution hardware accelerator based on FPGA. Background Art

[0002] At present, Convolutional Neural Network (CNN) has performed well in image classification, target detection, natural language processing, speech recognition, game theory and other fields. However, due to its computationally intensive and storage intensive characteristics, it encounters many difficulties when deployed on mobile devices or embedded devices. For example, there is a large demand for using CNN for information processing in the new generation of aerospace models, but the heat dissipation conditions of aerospace equipment are harsh, and the real-time requirements for computing are very high. In the past, the information processing methods based on general-purpose processors were increasingly difficult to meet the application requirements of aerospace models. For example, CPUs and DSPs are difficult to meet real-time requirements due to insufficient computing power, and GPUs are difficult to adapt to harsh cooling environments due to excessive power consumption. The general solution to the above problems is to design a special acceleration unit for CNN: design appropriate on-chip cache and data flow sorting methods based on the algorithm characteristics of CNN to improve the parallelism of computing and the reuse rate of data, while reducing the amount of off-chip memory access, thereby improving the overall computing power and energy efficiency of the system. Since the computational workload of the convolutional layer often accounts for more than 95% of the total computational workload of CNN, the design level of the hardware acceleration circuit directly determines the overall performance of the system. . Summary of the invention

[0003] In view of the above analysis, the present invention aims to disclose a two-dimensional multi-channel convolution hardware accelerator based on FPGA, which improves the computational efficiency of convolution layers of any scale while ensuring accuracy, thereby solving the real-time and power consumption problems encountered when deploying CNN in embedded devices.

[0004] The present invention discloses a two-dimensional multi-channel convolution hardware accelerator based on FPGA, comprising: a control unit and a bias cache, a weight cache, an input feature cache, a convolution cache, a weight pre-reading register group, a PE array, a nonlinear unit, a second gate S2 and a third gate S3 connected to and controlled by the control unit;

[0005] The feature cache is connected to the PE array; the weight cache is connected to the PE array through the weight pre-read register group; the bias cache and the convolution cache are connected to the PE array input through the third gate S3, and the PE array output is connected to the convolution cache and the nonlinear unit through the second gate S2;

[0006] Under the control of the control unit, the input feature cache, bias cache and weight cache are loaded with data; the weight pre-read register group pre-reads and stores the weight cache; the PE array writes the input features, the pre-read and stored weight data, and the bias data or the convolution intermediate result written through the third selector S3 to perform a convolution operation; and the convolution intermediate result is written into the convolution cache through the second selector S2, and the final convolution result is output after being activated by the nonlinear unit.

[0007] Furthermore, the PE array includes n parallel PE modules; each of the PE modules includes a separate weight register, a multiplier group, an adder, a convolution register, a shifter, an adder and a sum register; all the PE modules share an input feature register;

[0008] The weight register is a register with a bit width of m, an input end of which is connected to the weight pre-reading register group, and an output end of which is connected to the input end of the multiplier group;

[0009] The input feature register is a register with a bit width of m, an input end is connected to the input feature buffer, and an output end is connected to the input end of the multiplier group of all PE modules;

[0010] The multiplier group multiplies the input feature data and weight data in pairs, and outputs the multiplication results to the input end of the addition tree;

[0011] After the addition tree adds and sums the multiplication results, the summation result is output to an input terminal of the adder;

[0012] The convolution register stores the bias data or the convolution intermediate data output by the bias buffer or the convolution buffer, and then outputs the data to the other input end of the adder after passing through the shifter.

[0013] The adder adds the input summation result and the shifter output data bit by bit and outputs them to the sum register. The sum register outputs the bit-by-bit addition result to the convolution buffer or the nonlinear unit after passing through the second selector S2.

[0014] Furthermore, the number n of the PE modules is equal to the output parallelism of the convolution kernel, and the bit width m is equal to the input parallelism of the convolution kernel; the size of the weight pre-reading register group is m×n.

[0015] Furthermore, the input features, output features, weights, and bias data of the PE module are quantized to 8-bit fixed-point numbers;

[0016] In the multiplier group, two multiplications with different weights but the same input characteristics are combined together through the multiplication associative law and calculated using the large-bit-width DSP in the FPGA;

[0017] The product o is expressed as o=o1+o2=w1×a+(w2×a)<<16=(w1+w2<<16)×a; o1 is the product of feature data a and weight data w1, and o2 is the product of feature data a and weight data w2; << is shifted to the left;

[0018] The product o1 is the lower 16 bits of the product o. By truncation, we can get o1=w1×a=o from the product o. (15:0) ;

[0019] The product o2 is the high 16 bits of the product o, product o2 = w2 × a = (o>>16) + o

[15] ;>> is shifted to the right, o

[15] The highest bit of the product o.

[0020] Furthermore, the theoretical bit width of the convolution result output by the PE array is calculated to be 16+log2(M×K×K), where K×K is the size of the convolution kernel and M is the number of input channels of the convolution layer. The bit width of the convolution cache is obtained based on the theoretical bit width combined with the random number module test. The intermediate data and the final result of the convolution are saturated to prevent calculation result errors in extreme cases.

[0021] Furthermore, the bit width of the convolution buffer is 24 bits;

[0022] The saturation processing of the intermediate result of convolution is Conv is the intermediate result of convolution, which is stored in a convolution buffer with a bit width of 24 bits.

[0023] Furthermore, the final result of the convolution is represented by an 8-bit integer;

[0024] The saturation processing of the final result of convolution is Fout is the final result of the convolution, round() is the rounding operation; Fout_shif is the number of bits of the output feature shifted to the right.

[0025] Furthermore, the weight data is pre-sorted and loaded sequentially during calculation; the pre-sorted weight data is represented as a six-dimensional tensor Weights[N / n][M / m][K][K][n][m]; and is stored continuously in an external DDR in order from low dimension to high dimension, where M is the number of input channels of the convolution layer, N is the number of output channels of the convolution layer, and the size of the convolution kernel is K×K; m is the input channel parallelism of the two-dimensional multi-channel convolution hardware accelerator, and n is the output channel parallelism of the two-dimensional multi-channel convolution hardware accelerator.

[0026] Furthermore, the input feature data is represented as a five-dimensional tensor:

[0027] Fin[M / m][Hin / Tensor_Hin][Tensor_Hin][Win][m];

[0028] The output feature data is represented as a five-dimensional tensor:

[0029] Fout[N / n][Hout / Tensor_Hout][Tensor_Hout][Wout][n];

[0030] The input feature data and output feature data are stored continuously in the external DDR in the order from low dimension to high dimension; Win and Hin are the width and height of the input feature map; Wout and Hout are the width and height of the output feature map; Tensor_Hin and Tensor_Hout are the block sizes of the input and output feature maps in the height direction.

[0031] Furthermore, the acceleration process of the two-dimensional multi-channel convolution hardware accelerator includes:

[0032] 1) The control unit configures the current convolution layer parameters by receiving external instructions;

[0033] 2) The control unit controls the input feature cache, the bias cache and the weight cache to load the input feature, bias and weight data respectively; and while loading data into the weight cache, the data in the weight cache is written into the weight pre-read register group to preload the weight matrix;

[0034] 3) After the weight matrix is ​​preloaded, the weight registers of the PE array, the convolution registers, and the input feature registers are loaded from the weight pre-read register group, the bias cache, and the input feature cache at the same time within one clock cycle;

[0035] 4) After loading, the PE array performs convolution operations and stores the calculation results in the convolution buffer;

[0036] 5) Continue to load the input feature registers of the PE array from the input feature cache; repeat step 4) until the weight data in the convolution operation of the PE array is reused Tensor_Hout*Wout times, and then update the weight prefetch register group;

[0037] 6) After the weight pre-fetch register group is updated, within one clock cycle, the weight registers of the PE array, the convolution registers, and the input feature registers are loaded from the updated weight pre-read register group, the convolution buffer, and the input feature buffer at the same time;

[0038] 7) After loading is completed, the PE array performs convolution operation; and stores the calculation result into the convolution buffer through the second gate S2;

[0039] 8) Continue to load the convolution registers and input feature registers of the PE array from the convolution buffer and the input feature buffer; repeat step 7) until the weight data in the PE array is reused Tensor_Hout*Wout times, and update the weight prefetch register group again;

[0040] 9) Repeat steps 5)-8) until the (K×K×M / m)-1th group of convolution kernels is updated, the control unit controls the second gate S2 to gate the nonlinear unit; the PE array outputs the final result of the convolution to the nonlinear unit for activation operation and then output;

[0041] 10) Repeat steps 3)-9) until the convolution processing of the convolution layer is completed.

[0042] The present invention can achieve at least one of the following beneficial effects:

[0043] The present invention utilizes FPGA devices to realize efficient calculation of convolutional layers of any scale in CNN, thereby improving the versatility, efficiency, configurability and reliability of this part of the hardware circuit, and further improving the calculation real-time performance and energy consumption ratio of CNN. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0045] Figure 1 Schematic diagram of the calculation process of the convolutional layer in an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of the connection components of a two-dimensional multi-channel convolution hardware accelerator based on FPGA in an embodiment of the present invention;

[0047] Figure 3 The figure is a schematic diagram of the internal structure and connection of the PE module in the embodiment of the present invention;

[0048] Figure 4 The figure is a schematic diagram of the overall calculation process of the convolutional layer hardware implementation in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used to illustrate the principles of the present invention together with the embodiments of the present invention.

[0050] An embodiment of the present invention discloses a two-dimensional multi-channel convolution hardware accelerator based on FPGA to accelerate the convolution of CNN network;

[0051] like Figure 1As shown in the figure (the process of adding the convolution result and the bias data is omitted in this figure), the calculation process of the convolution layer is that multiple convolution kernels regularly scan the input feature map Fin to obtain a set of output feature maps Fout, that is, one convolution kernel corresponds to one output feature map. The convolution process can be abstracted as expanding the output channel loop and the input channel loop for parallel calculation. The remaining loops still maintain a compact nested structure, and adjust the nesting order according to the continuity of memory access. Finally, the convolution calculation of the standard convolution layer is completed through the nested loop body.

[0052] In this embodiment, the parameters of the convolution layer are set as follows: the number of input channels is M, the number of output channels is N, the width and height of the input feature map are Win and Hin respectively, the width and height of the output feature map are Wout and Hout respectively, the size of the convolution kernel is K×K, the convolution step is S, the padding width around the feature map is P, and the block sizes of the input and output feature maps in the height direction are Tensor_Hin and Tensor_Hout respectively. In addition, the input and output channel parallelism of the two-dimensional multi-channel convolution hardware accelerator are set to m and n respectively. Both m and n can be parameterized, and typical configurations are 8×8, 16×16, 32×32, etc.

[0053] According to the above settings and the calculation process of the convolution layer, the two-dimensional multi-channel convolution hardware accelerator based on FPGA designed in this embodiment is as follows: Figure 2 As shown, it includes: a control unit and a bias cache, a weight cache, an input feature cache, a convolution cache, a weight pre-read register group, a PE array, a nonlinear unit, a second gate S2 and a third gate S3 connected to and controlled by the control unit;

[0054] The feature cache is connected to the PE array; the weight cache is connected to the PE array through the weight pre-read register group; the bias cache and the convolution cache are connected to the PE array input through the third gate S3, and the PE array output is connected to the convolution cache and the nonlinear unit through the second gate S2;

[0055] Under the control of the control unit, the input feature cache, bias cache and weight cache are loaded with data; the weight pre-read register group pre-reads and stores the weight cache; the PE array writes the input features, the pre-read and stored weight data, and the bias data or the convolution intermediate result written through the third selector S3 to perform a convolution operation; and the convolution intermediate result is written into the convolution cache through the second selector S2, and the final convolution result is output after being activated by the nonlinear unit.

[0056] Preferably, it also includes a first selector S1; the input end of the first selector S1 is connected to the bias and weight data, and under the control of the control unit, the bias data is output to the bias cache, and the weight data is output to the weight cache.

[0057] Since the input feature map may be large, especially the initial layers of CNN include large input feature maps, the scale of convolution is also large, and the convolution calculation of the two-dimensional multi-channel convolution hardware accelerator cannot complete the convolution of the entire input feature map in one time; therefore, the input feature map and the output feature map are divided into blocks and indexed; by convolving the blocks relative to the feature maps in the blocks, and realizing the convolution of all feature maps according to the index, it can adapt to convolution layers of any scale.

[0058] Among them, the index of the input feature map is rin, cin, and the index of the output feature map is rout, cout; they are obtained by dynamic calculation based on various parameters.

[0059] The specific calculation method is:

[0060] rout=Tensor_Hout×row_id+row;

[0061] cout = col;

[0062]

[0063]

[0064] Among them, the number of input channels M, the number of output channels N, the height Hin of the input feature map, the width Win of the input feature map, the convolution step S, the feature map expansion width P, the size K of the convolution kernel, the row block size Tensor_Hout of the output feature map and other parameters can all be configured at runtime. Therefore, according to, the present invention can adapt to convolution layers of any scale.

[0065] Specifically, the PE array includes n parallel PE modules.

[0066] like Figure 3 As shown, each of the PE modules includes a separate weight register, a multiplier group, an adder, a convolution register, a shifter, an adder, and a sum register; all PE modules share an input feature register;

[0067] The weight register is a register with a bit width of m, an input end of which is connected to the weight pre-reading register group, and an output end of which is connected to the input end of the multiplier group;

[0068] The input feature register is a register with a bit width of m, an input end is connected to the input feature buffer, and an output end is connected to the input end of the multiplier group of all PE modules;

[0069] The multiplier group multiplies the input feature data and weight data in pairs, and outputs the multiplication results to the input end of the addition tree;

[0070] After the addition tree adds and sums the multiplication results, the summation result is output to an input terminal of the adder;

[0071] The convolution register stores the bias data or the convolution intermediate data output by the bias buffer or the convolution buffer, and then outputs the data to the other input end of the adder after passing through the shifter.

[0072] The adder adds the input summation result and the shifter output data bit by bit and outputs them to the sum register. The sum register outputs the bit-by-bit addition result to the convolution buffer or the nonlinear unit after passing through the second selector S2.

[0073] Specifically, the number of PE modules is the output channel parallelism of the PE array. Therefore, the number of PE modules is n, the bit width of each data processed by the PE module is the input channel parallelism of the PE array, and the bit width of each data processed by the PE module is m; in order to achieve the calculation data throughput coordination with the PE module, the bit width of the bias cache, weight cache and input feature cache are all m; the bit width of the convolution cache is n; the size of the weight pre-read register group is m×n.

[0074] A specific solution in this embodiment adopts the following specific measures to save the scarce DSP resources on the FPGA chip:

[0075] First, the input and output features, weights, and biases are quantized into 8-bit fixed-point numbers. Then, the associative law of multiplication is used to combine two multiplications with different weights but the same input features and use a large-bit-width DSP for calculation.

[0076] The specific process can be expressed as the following formula:

[0077] o=o1+o2=w1×a+(w2×a)<<16=(w1+w2<<16)×a;

[0078] Among them, o1 is the product of feature data a and weight data w1, o2 is the product of feature data a and weight data w2; << means shifting to the left.

[0079] The result o1 of a×w1 and the result o2 of a×w2 can be analyzed from the product o of the above formula in the following ways:

[0080] o1=w1×a=o (15:0)

[0081]

[0082] That is, o1 is the lower 16 bits of o, which can be parsed from o by truncation. o2 is the upper 16 bits of o, and the parsing process is divided into two cases: Case 1, when o1 is a positive number, only the lower 16 bits of o are affected, and o2 can be directly obtained by shifting; Case 2, when o1 is a negative number, 1 will be borrowed from the upper 16 bits, and at this time, 1 needs to be added to the shift result of o to compensate.

[0083] In order to omit the judgment on whether o1 is positive or negative, the parsing process in this embodiment is equivalent to the following process:

[0084] The product o1 is the lower 16 bits of the product o. By truncation, we can get o1=w1×a=o from the product o. (15:0) ;

[0085] The product o2 is the high 16 bits of the product o, product o2 = w2 × a = (o>>16) + o

[15] ;>> is shifted to the right, o

[15] The highest bit of the product o.

[0086] In the scheme of this embodiment, the setting of the convolution cache takes into account the situation of insufficient hardware cache space. The input features and weight data are cached in blocks on the chip for calculation, and the general process of convolution is sequentially adjusted and expanded in parallel, so that the calculation process of the output feature Fout is divided into multiple accumulation stages. The accumulation process will generate a large amount of intermediate result data, all of which are cached in the convolution cache.

[0087] For the theoretical analysis of the convolution output features, the calculation formula of the convolution output features is as follows:

[0088]

[0089] It can be seen from the above formula that the final result of the output feature requires M×K×K multiplication and accumulation operations.

[0090] When the bit width of the input feature and weight data is 8 bits, the theoretical bit width of the convolution result is 16+log2(M×K×K) bits.

[0091] Taking into account hardware storage resources and actual application scenarios, the bit width of the convolution buffer is obtained based on the theoretical bit width and random number module testing;

[0092] The convolution intermediate data and the final convolution result are saturated to prevent calculation result errors in extreme cases.

[0093] Through random number simulation tests, it is found that the bit width of 24 bits can basically meet the requirements.

[0094] In order to prevent the occurrence of extreme cases that may lead to incorrect calculation results, the specific method of saturation processing of the convolution intermediate process data is as follows:

[0095] The saturation processing of the intermediate result of convolution is Conv is the intermediate result of convolution, which is stored in a convolution buffer with a bit width of 24 bits.

[0096] The final result of the convolution is still represented by an 8-bit integer;

[0097] The saturation processing of the final result of convolution is Fout is the final result of the convolution, round() is the rounding operation; Fout_shif is the number of bits of the output feature shifted to the right.

[0098] Generally, a convolutional layer in a CNN is followed by a nonlinear activation layer, including different activation methods such as ReLU and Leaky-ReLU. In order to improve the computational efficiency of the system, the present invention integrates the calculation into the convolution circuit, and directly performs nonlinear operations after the final result of the convolution is calculated. The calculation method of Leaky-ReLU can be expressed as:

[0099] f(x)=max(α×x,x);

[0100] The hyperparameter ɑ is usually a very small constant, and its value range is between 0 and 1. In order to simplify the hardware circuit as much as possible, in this embodiment, the value of ɑ is limited to 1 / 2n, where n is a natural number, so that the floating-point multiplication operation of ɑ×x is simplified to a shift operation.

[0101] Since the input and output channel parallelism of the two-dimensional multi-channel convolution hardware accelerator are set to m and n respectively, in order to improve the continuity of loading weight data from the external storage DDR, the embodiment of the present invention chooses to pre-sort the weight data and load them sequentially during calculation.

[0102] The weight data before pre-sorting can be represented as a four-dimensional tensor as shown below: Weights[K][K];

[0103] The weight data is pre-sorted and loaded sequentially during calculation; the pre-sorted weight data is represented as a six-dimensional tensor Weights[N / n][M / m][K][K][n][m]; and is stored continuously in an external DDR in order from low dimension to high dimension, where M is the number of input channels of the convolution layer, N is the number of output channels of the convolution layer, and the size of the convolution kernel is K×K; m is the input channel parallelism of the two-dimensional multi-channel convolution hardware accelerator, and n is the output channel parallelism of the two-dimensional multi-channel convolution hardware accelerator.

[0104] In order to improve the continuity of loading weight data from the external storage DDR, the embodiment of the present invention chooses to pre-sort the input and output feature data and load them sequentially during calculation.

[0105] The general ordering of input and output features can be expressed as:

[0106] Fin[M][Hin][Win]

[0107] Fout[N][Hout][Wout]

[0108] After adjusting the sorting method, the input and output feature data can be represented as five-dimensional tensors as shown below, and are stored continuously in the external DDR in order from low dimension to high dimension for use by the convolution acceleration circuit during calculation.

[0109] The input feature data is represented as a five-dimensional tensor:

[0110] Fin[M / m][Hin / Tensor_Hin][Tensor_Hin][Win][m];

[0111] The output feature data is represented as a five-dimensional tensor:

[0112] Fout[N / n][Hout / Tensor_Hout][Tensor_Hout][Wout][n];

[0113] The input feature data and output feature data are stored continuously in the external DDR in the order from low dimension to high dimension; Win and Hin are the width and height of the input feature map; Wout and Hout are the width and height of the output feature map; Tensor_Hin and Tensor_Hout are the block sizes of the input and output feature maps in the height direction.

[0114] Preferably, the degree of parallelism is set to m=n. Therefore, it is only necessary to pre-sort the input features of the first layer of the CNN algorithm, and the subsequent convolutional layers all meet the condition that "the sorting method of the output features of the previous layer is the same as the sorting method of the input of the next layer", and no sorting process is required. The two dimensions [Hin / Tensor_Hin] and [Tensor_Hin] in the height direction are adjacent, so changing the block size in the height direction of the feature map during the convolution calculation process will not change the data sorting method.

[0115] Specifically, under the control of the control unit, the acceleration process of the two-dimensional multi-channel convolution hardware accelerator includes:

[0116] 1) The control unit configures the current convolution layer parameters by receiving external instructions;

[0117] The parameters include the number of input channels, the number of output channels, the size of the input feature map, the size of the output feature map, the convolution step size, the edge zero padding width, the convolution kernel size, the block parameters of the feature map and the grouping method of the convolution kernel;

[0118] 2) The control unit controls the input feature cache, the bias cache, and the weight cache to load the input feature, bias, and weight data respectively; and while loading data into the weight cache, the data in the weight cache is written into the weight pre-read register group to pre-load the weight matrix until the weight pre-read register group is full;

[0119] The stop and start of the loading process and the location (address) of the write cache are all controlled by the control unit.

[0120] 3) After the weight matrix is ​​preloaded, the weight registers of the PE array, the convolution registers, and the input feature registers are loaded from the weight pre-read register group, the bias cache, and the input feature cache at the same time within one clock cycle;

[0121] Weight register loading, which reads m×n data from the weight pre-read register group and writes it into the weight register in the PE array;

[0122] Convolution register loading: The first convolution register loading is to read the bias data from the bias buffer and write it into the convolution register in the PE array under the control of the third selector S3;

[0123] Input feature register loading is to read data from the input feature cache and write it to the input feature register in the PE array.

[0124] 4) After loading is completed, the PE array performs convolution operation; and stores the calculation result into the convolution buffer through the second gate S2;

[0125] Each PE module performs convolution calculations based on the data in each register and outputs the calculation results;

[0126] 5) Continue to load the input feature registers of the PE array from the input feature cache; repeat step 4) until the weight data in the convolution operation of the PE array is reused Tensor_Hout*Wout times, and then update the weight prefetch register group;

[0127] Furthermore, the update of the weight prefetch register group must be completed before the last calculation of the PE array before the update; otherwise, the PE array needs to wait until the update of the weight prefetch register group is completed before the next calculation;

[0128] 6) After the weight pre-fetch register group is updated, within one clock cycle, the weight registers of the PE array, the convolution registers, and the input feature registers are loaded from the updated weight pre-read register group, the convolution buffer, and the input feature buffer at the same time;

[0129] Under the control of the third selector S3, the convolution register of the PE module is connected to the convolution buffer, and the convolution register loads the convolution intermediate result from the convolution buffer;

[0130] 7) After loading is completed, the PE array performs convolution operation; and stores the calculation result into the convolution buffer through the second gate S2;

[0131] 8) Continue to load the convolution registers and input feature registers of the PE array from the convolution buffer and the input feature buffer; repeat step 7) until the weight data in the PE array is reused Tensor_Hout*Wout times, and update the weight prefetch register group again;

[0132] 9) Repeat steps 5)-8) until the (K×K×M / m)-1th group of convolution kernels is updated, the control unit controls the second gate S2 to gate the nonlinear unit; the PE array outputs the final result of the convolution to the nonlinear unit for activation operation and then output;

[0133] 10) Repeat steps 3)-9) until the convolution processing of the convolution layer is completed.

[0134] Under the control of the control unit, the acceleration process of the two-dimensional multi-channel convolution hardware accelerator can be expressed as follows: Figure 4 The execution process of the pseudo code diagram of 8 layers of tight nesting and sequential execution is shown.

[0135] In summary, the present invention uses FPGA devices to achieve efficient calculation of convolutional layers of any scale in CNN, improves the versatility, efficiency, configurability and reliability of this part of the hardware circuit, and further improves the real-time calculation and energy consumption ratio of CNN.

[0136] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A two-dimensional multi-channel convolution hardware accelerator based on FPGA, characterized in that: include: A control unit and a bias cache, a weight cache, an input feature cache, a convolution cache, a weight pre-read register group, a PE array, a nonlinear unit, a second gate S2 and a third gate S3 connected to and controlled by the control unit; The feature cache is connected to the PE array; the weight cache is connected to the PE array through the weight pre-read register group; the bias cache and the convolution cache are connected to the PE array input through the third gate S3, and the PE array output is connected to the convolution cache and the nonlinear unit through the second gate S2; Under the control of the control unit, the input feature cache, bias cache and weight cache are loaded with data; the weight pre-read register group pre-reads and stores the weight cache; the PE array writes the input features, the pre-read and stored weight data, and the bias data or the convolution intermediate result written through the third selector S3 to perform a convolution operation; and the convolution intermediate result is written into the convolution cache through the second selector S2, and the final convolution result is output after being activated by the nonlinear unit.

2. The two-dimensional multi-channel convolution hardware accelerator according to claim 1, characterized in that: The PE array includes n parallel PE modules; each of the PE modules includes a separate weight register, a multiplier group, an adder, a convolution register, a shifter, an adder and a sum register; all PE modules share an input feature register; The weight register is a register with a bit width of m, an input end of which is connected to the weight pre-reading register group, and an output end of which is connected to the input end of the multiplier group; The input feature register is a register with a bit width of m, an input end is connected to the input feature buffer, and an output end is connected to the input end of the multiplier group of all PE modules; The multiplier group multiplies the input feature data and weight data in pairs, and outputs the multiplication results to the input end of the addition tree; After the addition tree adds and sums the multiplication results, the summation result is output to an input terminal of the adder; The convolution register stores the bias data or the convolution intermediate data output by the bias buffer or the convolution buffer, and then outputs the data to the other input end of the adder after passing through the shifter. The adder adds the input summation result and the shifter output data bit by bit and outputs them to the sum register. The sum register outputs the bit-by-bit addition result to the convolution buffer or the nonlinear unit after passing through the second selector S2.

3. The two-dimensional multi-channel convolution hardware accelerator according to claim 2, characterized in that: The number n of the PE modules is equal to the output parallelism of the convolution kernel, and the bit width m is equal to the input parallelism of the convolution kernel; the size of the weight pre-reading register group is m×n.

4. The two-dimensional multi-channel convolution hardware accelerator according to claim 2, characterized in that: The input features, output features, weights, and bias data of the PE module are all quantized to 8-bit fixed-point numbers; In the multiplier group, two multiplications with different weights but the same input characteristics are combined together through the multiplication associative law and calculated using the large-bit-width DSP in the FPGA; The product o is expressed as o=o1+o2=w1×a+(w2×a)<<16=(w1+w2<<16)×a; o1 is the product of feature data a and weight data w1, and o2 is the product of feature data a and weight data w2; << is shifted to the left; The product o1 is the lower 16 bits of the product o. By truncation, we can get o1=w1×a=o from the product o. (15:0) ; The product o2 is the high 16 bits of the product o, product o2 = w2 × a = (o>>16) + o [15] ;>> is shifted to the right, o [15] The highest bit of the product o.

5. The two-dimensional multi-channel convolution hardware accelerator according to claim 2, characterized in that: The theoretical bit width of the convolution result output by the PE array is calculated to be 16+log2(M×K×K), where K×K is the size of the convolution kernel and M is the number of input channels of the convolution layer. The bit width of the convolution cache is obtained based on the theoretical bit width combined with the random number module test. The intermediate data and the final result of the convolution are saturated to prevent calculation errors in extreme cases.

6. The two-dimensional multi-channel convolution hardware accelerator according to claim 5, characterized in that: The bit width of the convolution buffer is 24 bits; The saturation processing of the intermediate result of convolution is Conv is the intermediate result of convolution, which is stored in a convolution buffer with a bit width of 24 bits.

7. The two-dimensional multi-channel convolution hardware accelerator according to claim 6, characterized in that: The final result of the convolution is represented by an 8-bit integer; The saturation processing of the final result of convolution is Fout is the final result of the convolution, round() is the rounding operation; Fout_shif is the number of bits of the output feature shifted to the right.

8. The two-dimensional multi-channel convolution hardware accelerator according to claim 1, characterized in that: The weight data is pre-sorted and loaded sequentially during calculation; the pre-sorted weight data is represented as a six-dimensional tensor Weights[N / n][M / m][K][K][n][m]; and is stored continuously in an external DDR in order from low dimension to high dimension, where M is the number of input channels of the convolution layer, N is the number of output channels of the convolution layer, and the size of the convolution kernel is K×K; m is the input channel parallelism of the two-dimensional multi-channel convolution hardware accelerator, and n is the output channel parallelism of the two-dimensional multi-channel convolution hardware accelerator.

9. The two-dimensional multi-channel convolution hardware accelerator according to claim 8, characterized in that: The input feature data is represented as a five-dimensional tensor: Fin[M / m][Hin / Tensor_Hin][Tensor_Hin][Win][m]; The output feature data is represented as a five-dimensional tensor: Fout[N / n][Hout / Tensor_Hout][Tensor_Hout][Wout][n]; The input feature data and output feature data are stored continuously in the external DDR in the order from low dimension to high dimension; Win and Hin are the width and height of the input feature map; Wout and Hout are the width and height of the output feature map; Tensor_Hin and Tensor_Hout are the block sizes of the input and output feature maps in the height direction.

10. The two-dimensional multi-channel convolution hardware accelerator according to claim 9, characterized in that: The acceleration process of the two-dimensional multi-channel convolution hardware accelerator includes: 1) The control unit configures the current convolution layer parameters by receiving external instructions; 2) The control unit controls the input feature cache, the bias cache and the weight cache to load the input feature, bias and weight data respectively; and while loading data into the weight cache, the data in the weight cache is written into the weight pre-read register group to preload the weight matrix; 3) After the weight matrix is ​​preloaded, the weight registers of the PE array, the convolution registers, and the input feature registers are loaded from the weight pre-read register group, the bias cache, and the input feature cache at the same time within one clock cycle; 4) After loading is completed, the PE array performs convolution operation; and stores the calculation result into the convolution buffer through the second gate S2; 5) Continue to load the input feature registers of the PE array from the input feature cache; repeat step 4) until the weight data in the convolution operation of the PE array is reused Tensor_Hout*Wout times, and then update the weight prefetch register group; 6) After the weight pre-fetch register group is updated, within one clock cycle, the weight registers of the PE array, the convolution registers, and the input feature registers are loaded from the updated weight pre-read register group, the convolution buffer, and the input feature buffer at the same time; 7) After loading is completed, the PE array performs convolution operations and stores the calculation results in the convolution buffer; 8) Continue to load the convolution registers and input feature registers of the PE array from the convolution buffer and the input feature buffer; repeat step 7) until the weight data in the PE array is reused Tensor_Hout*Wout times, and update the weight prefetch register group again; 9) Repeat steps 5)-8) until the (K×K×M / m)-1th group of convolution kernels is updated, the control unit controls the second gate S2 to gate the nonlinear unit; the PE array outputs the final result of the convolution to the nonlinear unit for activation operation and then output; 10) Repeat steps 3)-9) until the convolution processing of the convolution layer is completed.

Images