A method and device for implementing quantization of a convolutional neural network based on an FPGA

Abstract

The application discloses a kind of based on FPGA's convolutional neural network quantization implementation method and device, the quantization implementation method includes quantization CNN convolution kernel step, comprising: the CNN convolution layer is decomposed into the combination of logic operation and addition operation, is converted into N×N convolution logic kernel;In FPGA, a plurality of LUT is used to realize logic operation and addition operation respectively, to realize N×N convolution logic kernel.The application can make full use of the powerful logic operation capability of FPGA, and efficiently realizes CNN network low-bit quantization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and particularly relates to a convolutional neural network quantization implementation method and device based on FPGA. BACKGROUND

[0002] CNN (Convolutional Neural Networks) is a kind of feedforward neural network containing convolution calculation and having a deep structure, and is one of the representative algorithms of deep learning. On the hardware level, the current deployment of CNN is mainly based on CPU, GPU, ASIC and FPGA, and each type of deployment has its own advantages and disadvantages:

[0003] 1. Based on CPU

[0004] The operation of CPU is serial, and it has the characteristics of high flexibility and good portability, but it has too large delay in large-scale parallel operation, and it is difficult to apply to large-scale CNN network, and can only be applied to some specially designed small-scale network.

[0005] 2. Based on GPU

[0006] The flexibility of GPU is slightly worse than that of CPU, but GPU has many computing cores, which also makes it have stronger parallel processing capability, and it also has stronger ability to control data flow and store data. But the power consumption of GPU is too large, which limits its use on some embedded devices.

[0007] 3. Based on ASIC

[0008] ASIC has the lowest power consumption and has cost advantage in mass production. The customization characteristics of ASIC determine that its portability is low, its development cost is high, and its production cycle is long, which also limits its development in the current network and data rapid iteration.

[0009] 4. Based on FPGA

[0010] FPGA has achieved a trade-off between GPU and ASIC, and has well balanced processing speed, power consumption and development cost. However, one disadvantage of FPGA is that it requires users to use hardware description language to program it, which is difficult to develop.

[0011] FPGA(Field Programmable Gate Array) is the product of further development on the basis of PAL, GAL, CPLD and other programmable logic devices. It is a kind of semi-custom circuit in the field of application-specific integrated circuit, which not only solves the shortcomings of full-custom circuit, but also overcomes the limitation of the number of gate circuits of the original programmable logic device. FPGA development is very different from traditional PC and single-chip microcomputer development. FPGA mainly uses parallel operation and hardware description language to achieve it, which is very different from the sequential operation of PC or single-chip microcomputer (whether it is von Neumann structure or Harvard structure). FPGA development needs to start from top-level design, module layering, logic implementation, software and hardware debugging and other aspects. FPGA can be repeatedly programmed by burning bit stream file. At present, most FPGAs use look-up table structure based on SRAM (StaTIc Random Access Memory) technology, and the content of the look-up table is changed to realize configuration by burning bit stream file. Compared with GPU and FPGA, although the flexibility and portability are poor and the development difficulty is high, FPGA has the advantages of low power consumption and high speed, and also has certain cost advantage, which is suitable for embedded terminal deployment under certain batch. Compared with ASIC, FPGA has more design space with increasing gate resources and memory bandwidth, and FPGA also saves the flow process required in ASIC scheme, with short development cycle and low development cost.

[0012] When deploying FPGA using FPGA, the CNN network needs to be decomposed into a structure suitable for FPGA implementation. The computing unit of FPGA is divided into DSP, multiplier-adder, LUT (logic lookup table), and each operation of CNN needs to be mapped to the corresponding operation logic according to the FPGA computing unit 1:1. On the FPGA side, the on-chip resources are used to integrate the units required for data carrying operation and data computing operation to form a hardware operation layer.

[0013] Network quantization is the key to the implementation of deep learning at the application end. At present, various deep learning frameworks have enabled quantization. Through network quantization, the following can be achieved:

[0014] 1. Reduce memory capacity. When the weights of the convolution layer are quantized from 32-bit floating-point numbers to 8-bit integers, the memory capacity of the weights is reduced to 1 / 4 of the original, which enables the edge to save more memory capacity and storage space.

[0015] 2. Reduce memory bandwidth. When the activation value of the convolution layer is quantized from 32-bit floating-point number to 8-bit integer, the memory bandwidth of the activation value when passing down can be reduced to 1 / 4 of the original, which is very helpful for reducing memory occupation and improving reading performance.

[0016] 3. Save computing resources. When performing convolution operation, when the weight and activation value are quantized by 32-bit floating point to 8-bit integer, the operation efficiency can be improved by more than 4 times on the device supporting 8-bit multiplication, which greatly saves the CPU operation resource.

[0017] 4. The specially designed quantization method makes it possible to implement the network on a low-end FPGA. FPGA has the advantages of low energy consumption and programmability. When the specially designed quantization network is implemented on FPGA, the algorithm can be hardwareized, saving cost.

[0018] Traditional floating point convolution or fixed point number convolution needs to rely on a large number of multiplication and addition, so when the CNN network quantization is implemented on FPGA, there will be a large amount of calculation, resulting in large amount of calculation and low efficiency of CNN implementation. The current neural network deployment on FPGA generally uses 8bW / 8bA or 4bW / 4bA quantization, which needs to be based on multiplication operation, and cannot fully utilize the powerful logic operation capability of FPGA, so the actual quantization efficiency is not high. The existing network quantization algorithms in the prior art need to implement low-bit quantization, such as Hash mapping, non-uniform quantization, and the implementation logic is not hardware-friendly, that is, it is not convenient to implement in hardware, and it is difficult to popularize to general scenarios, so it is almost impossible to introduce software stack, and it is not suitable for CNN deployment quantization on FPGA. SUMMARY

[0019] The technical problem to be solved by the present application is that in view of the technical problems existing in the prior art, the present application provides a convolutional neural network quantization implementation method and device based on FPGA, which can fully utilize the powerful logic operation capability of FPGA and efficiently implement low-bit quantization of CNN network.

[0020] To solve the above technical problems, the technical solution provided by the present application is:

[0021] A convolutional neural network quantization implementation method based on FPGA, the method comprising a quantization convolution kernel step, comprising:

[0022] The CNN convolution layer is decomposed into a combination of logic operation and addition operation, and is converted into an N*N convolution logic kernel, N is a positive integer greater than 2;

[0023] In the FPGA, a plurality of LUTs are used to implement the logic operation and the addition operation respectively, so as to implement the N*N convolution logic kernel.

[0024] Further, the N*N convolution logic kernel is obtained by splitting the quantized activation value in the CNN convolution layer according to the preset quantization bit number, and is decomposed into a combination of a plurality of exclusive OR operations and a plurality of addition operations; the N*N convolution logic kernel includes the exclusive OR operation between the converted quantized convolution parameter w' and the quantized activation value.

[0025] Further, when N=3, a 3x3 convolution logic kernel is implemented, and multiple LUTs are used to calculate wherein and are the 2nd, 1st, 0th bits of the quantized activation value, w'1, w'2, w'3 are the converted quantized convolution parameters corresponding to the bits, is the exclusive or symbol, a LUT is used to implement three two-digit addition, and an adder is used to implement the summation of each branch output.

[0026] Further, the method further comprises merging the convolution layer, the scale layer, and the activation layer to form a QCBR layer, wherein the QCBR layer is obtained by multiplying the N x N convolution logic kernel by a multiple coefficient and then adding a scale layer offset coefficient, so that the QCBR layer is implemented on the FPGA using logical operations, fixed-point multiplication, and fixed-point addition.

[0027] Further, the step of implementing the QCBR layer comprises:

[0028] using the N x N convolution logic kernel to calculate each N x N convolution on the input channel in the FPGA to obtain an N x N convolution result;

[0029] using an adder to accumulate the N x N convolution results on all input channels to obtain a convolution accumulation result;

[0030] using a multiplier-adder to multiply the convolution accumulation result by a corresponding preset multiple coefficient and then add a scale layer offset coefficient to obtain a scale operation result;

[0031] limiting the scale operation result and outputting.

[0032] Further, the method further comprises a training step, comprising:

[0033] constructing an initial CNN network N and training to obtain a pre-trained model BWN-model;

[0034] replacing the activation layer in the initial CNN network N with a preset activation layer QRelu to form an updated CNN network QN;

[0035] loading the weights of each layer of the pre-trained model BWN-model into the updated CNN network QN, and training the updated CNN network QN until the training is completed.

[0036] Further, the process of training the updated CNN network QN comprises updating the quantization coefficients of each activation layer QRelu in batches, and the steps comprise:

[0037] For each batch of data, forward inference is performed on the current network to determine the input of each activation layer QRelu in the updated CNN network QN. If there is no number greater than 0, the quantization coefficient is updated according to the current batch b, otherwise the standard deviation δ of [a j , -a j ] is calculated, a j is the number greater than 0 in the activation layer QRelu, and 2δ is the quantization mapping range. The quantization coefficient is updated according to the standard deviation δ and the current batch b.

[0038] A FPGA-based convolutional neural network quantization implementation device, comprising an FPGA, and a quantization convolution kernel module configured on the FPGA, wherein the quantization convolution kernel module comprises:

[0039] A first LUT unit comprising a plurality of LUTs, used to calculate logical operations in an N×N convolution logic kernel converted from a CNN convolution layer, wherein the N×N convolution logic kernel is converted from the CNN convolution layer by decomposing the CNN convolution layer into a combination of logical operations and addition operations;

[0040] A second LUT unit comprising a plurality of LUTs, used to calculate addition operations in the N×N convolution logic kernel converted from the CNN convolution layer;

[0041] An adder unit used to sum each branch to obtain a final result.

[0042] Further, the first LUT unit specifically calculates logical operations between weight parameters and quantized activation values of the 0th, 1st and 2nd bits in a 3×3 convolution logic kernel, the second LUT unit specifically calculates addition operations of logical operation results between weight parameters and quantized activation values of the 1st bit in the 3×3 convolution logic kernel, and the output end of the first LUT unit is further provided with a bit splicing circuit for bit splicing of each bit of data of each LUT 62, and the output end of the second LUT unit is further provided with a last bit 0 supplementing circuit for last bit 0 supplementing of output data of the second LUT unit.

[0043] Further, it further comprises a QCBR layer implementation module formed by fusing a convolution layer, a scale layer and an activation layer, wherein the QCBR layer implementation module comprises:

[0044] A quantization convolution unit used to calculate each N×N convolution on an input channel using the quantization convolution kernel module to obtain an N×N convolution result;

[0045] An accumulator used to accumulate the N×N convolution results on all input channels to obtain a convolution accumulation result; and multiply the convolution accumulation result by a corresponding preset fusion coefficient and add a scale layer offset coefficient to obtain a scale operation result.

[0046] A limiter is used to limit the scale operation result and output.

[0047] Compared with the prior art, the present application has the advantages that:

[0048] 1、The present application decomposes the CNN convolution layer into logical operation and addition operation, converts into N*N convolution logic kernel, and then realizes the N*N convolution logic kernel in FPGA, since only logical operation and addition operation are needed, it is suitable for FPGA implementation, can give full play to the powerful logical operation capability of FPGA, greatly reduces the required calculation amount, memory capacity and read-write bandwidth, and realizes efficient low-bit quantization CNN convolution layer.

[0049] 2、The present application further combines the convolution layer, the scale layer and the activation layer, and fuses to form a QCBR layer, the QCBR layer only contains simple logical operation, fixed-point multiplication and fixed-point addition, and is suitable for FPGA implementation, so that the calculation functions of the convolution layer, the scale layer and the activation layer can be efficiently realized at the same time by making full use of FPGA, and the required calculation amount, memory capacity and read-write bandwidth are further reduced.

[0050] 3、The present application further realizes low-bit quantization CNN network quantization by using the pre-quantization method of training and quantization at the same time, compared with the traditional quantization mode, the efficiency and precision of quantization training can be effectively improved, and high-precision training of low-bit quantization CNN network is realized. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flow schematic diagram of the FPGA-based convolutional neural network quantization implementation method of the embodiment.

[0052] Figure 2 is a structural principle schematic diagram of realizing 3*3 convolution logic kernel based on FPGA in the specific application embodiment of the present application.

[0053] Figure 3 is a structural principle schematic diagram of realizing QCBR layer based on FPGA in the embodiment.

[0054] Figure 4 is a flow schematic diagram of the traditional hybrid quantization mode. DETAILED DESCRIPTION

[0055] The present application will be further described below in combination with the drawings of the specification and specific preferred embodiments, but the protection scope of the present application is not limited by this.

[0056] The network model quantization of the CNN mainly includes two parts, one is quantization of weights, and the other is quantization of activation values, and when the weights and the activation values are quantized to 8 bits, the performance equivalent to 32 bits can be achieved. The basic operation in the neural network is convolution and multiplication-addition operation of the weights and the activation values, if one of them is quantized to { -1, 1}, the multiplication-addition operation can be simplified to addition-subtraction operation, and if both of them are quantized to { -1, 1}, the multiplication-addition operation can be simplified to bitwise operation. Through the above simplification of the CNN quantization to addition-subtraction operation and bitwise operation, the quantization process is friendly to hardware calculation, and is convenient for hardware implementation. Based on the above consideration, the CNN convolution layer is decomposed into logical operation and addition operation, converted into an N*N convolution logic core, N is the number of convolution kernels, and then the N*N convolution logic core is realized in the FPGA. Since only logical operation and addition operation are needed, the powerful logical operation capability of the FPGA can be fully utilized, and the efficient low-bit quantization CNN convolution layer is realized.

[0057] As shown in Figure 1 , the FPGA-based convolutional neural network quantization implementation method of the embodiment includes a quantized convolution kernel step, which includes:

[0058] S01. decomposing the CNN convolution layer into a combination of logical operation and addition operation, converting into an N*N convolution logic core, N is a positive integer greater than 2;

[0059] S02. using a plurality of LUTs (logical lookup tables) to respectively realize logical operation and addition operation in the FPGA to realize the N*N convolution logic core.

[0060] In the embodiment, the N*N convolution logic core is obtained by splitting the quantized activation values in the CNN convolution layer according to a preset quantization bit number, and decomposing into a combination of a plurality of exclusive OR operations and a plurality of addition operations.

[0061] When the quantization bit n=k, k is a positive integer greater than 1, the derivation process of the 3*3 convolution logic core converted from the quantized convolution formula is as follows:

[0062]

[0063]

[0064] wherein, x i，j is a quantized activation value, i represents any one channel from 1 to c, c is the number of input channels, j represents the jth value in the N*N block, w i，j is a weight coefficient, q in the formula represents an accumulated variable.

[0065] Taking N=3 and quantization bit depth as an example, the derivation process of converting the quantization convolution formula to obtain a 3×3 convolution logic kernel is as follows:

[0066]

[0067] In the above formula, w is the quantized convolution parameter with a value of [-1, 1], and c is the number of input channels; w′ is the transformed quantized convolution parameter with a value of [0, 1], which maps to the value of w; x is the 3-bit quantized activation value with a value of [0, 1, ..., 6, 7]. 2 The second quantized activation value is [0, 1]; x 1 The first bit of the quantized activation value has a value of [0, 1]; x 0 The 0th bit of the quantization activation value has a value of [0, 1].

[0068] In the above derivation process, the activation value x is first quantized. ij Bitwise decomposition representation, i.e. Because of w i，j The value is [-1, 1]. The value is [0, 1]. Multiplying them will result in a 3-value pair, which cannot be represented by a single bit. Therefore, [the value is...]. Transform into The result of multiplying after transformation is [-1, 1], which is still a value of 2; because The value is [-1, 1]. Since it cannot be represented by a single digit, it is determined using the formula... Will Performing equivalent substitutions, where ⊙ represents the XOR symbol, the resulting 3×3 convolutional logical kernel is: Because of w i，j Both c and are known numbers. Let be... Then β c This is a constant term and can be merged with subsequent scaling layers. The above 3×3 convolutional logic kernel is obtained based on the decomposition of 3-bit quantization bits. Each part only requires one bit for representation. In an FPGA, the above 1bW3bA 3×3 convolutional kernel logic can be implemented through LUTs and adders.

[0069] To implement the above 3×3 convolutional logic kernel f 3×3 (w′ i x i ),like Figure 2 As shown, multiple LUTs can be used for computation. in as well as These are the 2nd, 1st, and 0th bits of the quantized activation value, respectively. w′1, w′2, and w′3 are the quantized convolution parameters after transformation for the corresponding bits. A LUT is used to implement the addition of three two-digit numbers (outmh, outmm, outml), and an adder is used to sum the outputs of each branch (out1, out2, out3, out4).

[0070] See Figure 2 The 3×3 convolution logic kernel f is implemented using 22 LUTs (including 9 LUT62s and 1 LUT64, where each LUT62 consists of 2 LUTs and each LUT64 consists of 4 LUTs), several bit manipulation circuits, and adders. 3×3 (w′ i x i LUT is the smallest programmable unit in an FPGA, and the result of its operation is one bit of data. Figure 2 That is, using LUT to implement 3-bit features. Figure 1 A 3x3 convolution with bit weights, where arrive It is the least significant bit of the nine 3-bit feature data input. arrive It is the middle bit of the input 9 3-bit feature maps. arrive It is the highest bit of the nine 3-bit feature maps input, w′ i，1 to w′ i，9 These are the nine 1-bit weights input. The calculation is performed using a LUT62. The function is to perform logical operations on the weights wi,1-wi,9 and the 0th, 1st, and 2nd bits of the quantized activation values ​​xi,1-xi,9 respectively, and output a 2-bit value with a range of [0,3]. The LUT64 implements the addition function for three two-digit numbers (outmh, outmm, outml), where outmh, outmm, and outml are w′ respectively. i，1 to w′ i，9 and arrive The three logical operations result in a 4-bit output with a value range of [0, 9]. After multiplying by 2, the value range becomes [0, 18]. The adder sums the outputs of the four branches (out1, out2, out3, out4) to obtain a 3*3 convolution result with an output value range of [0, 63], which can be represented by a 6-bit unsigned integer. The bit concatenation circuit is set at the output of each LUT62 to combine the data bits together without occupying LUT resources. The trailing zero padding circuit is output at the output of LUT64 to append 1 bit 0 data to the end of the data, without occupying FPGA resources.

[0071] In addition to the above quantization of 3 bits, other quantization bit numbers (quantization bit numbers greater than or equal to 2) can also be used, that is, an n-bit activation value is represented as: x i The i-th bit of x is represented; then the same reasoning is used for the convolution formula, and different FPGA implementation methods can be obtained for different quantization bit numbers.

[0072] The following is an example of n = 4 quantization bits:

[0073]

[0074]

[0075] That is, when the quantization bit n = 4, the 3x3 convolution logic kernel f 3×3 (w′ i , x i ) is

[0076] In addition to the above-mentioned 3x3 convolution logic kernel, other NxN convolutions can also be constructed, with n-bit quantization, and the principle is the same as above, that is:

[0077]

[0078] where x ij is the quantized activation value, and w i，j is the weight coefficient.

[0079] Taking N = 5 as an example, the convolution logic kernel is derived as follows:

[0080]

[0081]

[0082] In summary, the present embodiment is to divide the CNN convolution layer into multiple logical operations and multiple addition operations according to the quantization bit n, and construct an NxN convolution logic kernel. Each operation part in the convolution logic kernel can be represented by 1 bit, so that the NxN convolution logic kernel can be efficiently implemented by LUT and adder in FPGA. By dividing the CNN network into a structure suitable for FPGA implementation, the required calculation amount, memory capacity and read-write bandwidth of the CNN can be greatly reduced. Taking a 3x3 convolution as an example, the calculation amounts of different quantization methods are shown in Table 1:

[0083] Table 1: Comparison of different quantization methods for a 3x3 convolution

[0084] Quantization manner Parameter size (bit) Activation value size (bit) Computational volume FP32 288 32 9 times of floating point multiplication + 9 times of floating point addition Int8 72 8 9 times of 8-bit multiplication + 9 times of 32-bit addition 1bW 3bA 9 3 22 times of logical operation + 4 times of 6-bit addition

[0085] As described above, the rest of the convolution formula after conversion is constant except for the convolution kernel, because it can be merged with the scale layer. The embodiment also includes merging the convolution layer, the scale layer, and the activation layer to form a QCBR layer, and using logical operations, fixed-point multiplication, and fixed-point addition to implement the QCBR layer on the FPGA, so that the operation performance of the FPGA can efficiently implement the functions of the convolution layer, the scale layer, and the activation layer.

[0086] To construct the QCBR layer, the embodiment first defines the following parameters for the QCBR layer:

[0087] (1) c: the number of input channels;

[0088] (2) l: the sequence number of the convolution layer;

[0089] (3) w: the quantized convolution parameter, whose value is [-1, 1];

[0090] (4) The parameter quantization coefficient of the lth convolution layer;

[0091] (5) x: the Uint3 quantized input activation value, which is the quantized output of the (l-1)th QCBR layer and the quantized input of the lth QCBR layer;

[0092] (6) The quantization coefficient of the output activation value of the (l-1)th QCBR layer, which is a single coefficient constant;

[0093] (7) The quantization coefficient of the output activation value of the lth QCBR layer, which is a single coefficient constant;

[0094] (8) The multiple coefficient of the lth scale layer;

[0095] (9) The offset coefficient of the lth scale layer;

[0096] (10) The multiple coefficient of the lth QCBR layer;

[0097] (11) The offset coefficient of the lth QCBR layer.

[0098] Then, the convolution layer, the scale layer, and the activation layer are fused to form the QCBR layer:

[0099] The convolution layer is represented as:

[0100]

[0101] The scale layer is represented as:

[0102]

[0103] The activation layer is represented as:

[0104]

[0105] The above (4), (5), (6) are fused to form:

[0106]

[0107] wherein,

[0108] The function is a limiting function, when x is less than a, the output is a; when x is greater than b, the output is b; otherwise, the output is x.

[0109] The above quantized convolution logic kernel is only an example of a 3x3 convolution logic kernel, and of course other NxN convolution logic kernels can also be used.

[0110] In the FPGA, the above formula (8) is implemented by a calculation unit, i.e., a QCBR layer is implemented, the QCBR layer is composed of a convolution logic kernel multiplied by a multiple coefficient and then added to a scale layer offset coefficient, and only contains simple logic element calculation, fixed-point multiplication and fixed-point addition, so that the calculation functions of the convolution layer, the scale layer and the activation layer can be efficiently implemented simultaneously by the FPGA.

[0111] In this embodiment, the steps of implementing the QCBR layer include:

[0112] In the FPGA, an NxN convolution logic kernel is used to calculate each NxN convolution on the input channel to obtain an NxN convolution result;

[0113] An adder is used to accumulate the NxN convolution results on all input channels to obtain a convolution accumulation result;

[0114] A multiply-accumulator is used to multiply the convolution accumulation result by a corresponding preset multiple offset coefficient and add a scale layer offset coefficient to obtain a scale operation result;

[0115] The scale operation result is limited and output.

[0116] Taking the QCBR layer as an example, in the FPGA, first, a quantized convolution logic kernel is used to calculate each 3x3 convolution on the input channel Then, an adder is used to accumulate the 3x3 convolution results on all input channels; then, a multiply-accumulator is used to multiply the convolution accumulation result by and add to obtain a scale operation result; finally, a comparator is used to limit and output the scale operation result.

[0117] Training quantization, also known as pre-quantization, involves quantizing the network during training. It offers high accuracy and model consistency, and low-order quantization networks must use training quantization. To implement the aforementioned low-order quantized CNN network in an FPGA, this embodiment employs a training quantization method. This training quantization method includes:

[0118] Construct an initial CNN network N and train it to obtain a pre-trained model BWN-model;

[0119] The activation layers in the initial CNN network N are replaced with preset QReLU layers to form the updated CNN network QN;

[0120] The weights of each layer of the pre-trained BWN-model are loaded into the updated CNN network QN, and the updated CNN network QN is trained until training is complete.

[0121] The activation value of the QRelu layer described above is defined according to the following formula:

[0122]

[0123] in q r These are the quantization coefficients for the QReLU layer, with an initial value of 6 / (2). q -1); a i a is the input activation value of the QReLU layer. k is the output activation value of the QReLU layer, and a and b are preset thresholds.

[0124] Traditional FP32 and INT8 hybrid quantization methods (such as...) Figure 4 As shown, FP32 is replaced by INT8 during inference, but training is still based on FP32, resulting in low training efficiency and accuracy. This embodiment uses a pre-quantization method that performs low-bit quantization of CNN networks while training, which can effectively improve the efficiency and accuracy of quantization training compared to the traditional hybrid quantization method of FP32 and INT8, achieving high-precision training of low-bit quantized CNN networks.

[0125] In this embodiment, the training process of the updated CNN network QN includes updating the quantization coefficients of each QReLU layer in batches. The specific steps include:

[0126] Each time, the data of the current batch is taken, forward inference is performed on the current network, and the input of each QReLU layer in the updated CNN network QN is judged. If there is no number greater than 0, the quantization coefficient is updated according to the current batch b; otherwise, [a] is calculated. j -a j The standard deviation δ, a jFor the number greater than 0 in the QRelu layer, 2d is the quantization mapping range, and the quantization coefficient is updated according to the standard deviation d and the current batch b.

[0127] In a specific application embodiment, the detailed steps of the above training method are as follows:

[0128] Step 1, construct an L-layer CNN network N, replace all Relu layers in the network with Relu6 layers, that is, use Relu6 as the activation function, and Relu6 is specifically represented as:

[0129]

[0130] Step 2, train the network N (the network training method can be selected according to actual needs, such as the xNOR-netBWN network training method) to obtain a pre-trained model BWN-model.

[0131] Step 3, define the activation value quantization bit number q, q is a positive integer greater than 1; replace the Relu6 layer in the network N with the QRelu layer to generate a new quantization network named QN, wherein the activation value of the QRelu layer is defined according to the above formula (8), and the gradient is defined as:

[0132]

[0133] Wherein, is the gradient of the input activation value of the QRelu layer, is the gradient of the output activation value of the QRelu layer.

[0134] Load the weights of each layer of the pre-trained model BWN-model into QN.

[0135] Step 4, define the total number of training rounds E, and set the current training round e to 0.

[0136] Step 5, update the quantization coefficient q of each QRelu layer r .

[0137] Step 5.1 Let B be the number of the smallest batch in a round, and set the current batch b to 0;

[0138] Step 5.2 Take the data of the current batch b, and perform forward inference on the network. For the input of each QRelu layer in the L-layer CNN quantization network QN, the following operations are performed:

[0139] Let l be a layer in the L-layer CNN quantization network, 1≤l≤L, and take the number greater than 0 in the input of the QRelu layer of the lth layer, denoted as a j . If the number of a j is 0, update Let Otherwise, calculate the standard deviation δ of [a j , -a j ] and update the quantization mapping range as 2δ. Let

[0140] Step 5.3, the current batch b is incremented by 1, if b is greater than or equal to B, jump to 6; otherwise, jump to 5.2.

[0141] Step 6, train the network QN.

[0142] Step 7, the current training round e is incremented by 1, if e is greater than or equal to E, end; otherwise, jump to 5.

[0143] The activation value quantization coefficient of the above QRelu layer can also not be based on a statistical manner, but a manner of directly specifying a quantization range, such as taking q r = P / (2 q -1), P is the quantization mapping range.

[0144] To verify the effectiveness of the above training method of the application, in a specific application embodiment, the traditional multiple different quantization network methods and the method of the application are compared for different types of networks, and the results are shown in Table 2. As can be seen from the table, through the special low-bit quantization network training method of the application, the precision of the low-bit quantization network can be only slightly lower than that of the full-precision network, that is, high-precision network quantization can be achieved.

[0145] Table 2: Comparison of indicators of different types of networks under different quantization methods

[0146]

[0147] The FPGA-based convolutional neural network quantization implementation device of the embodiment includes an FPGA, and a quantization convolution kernel module is configured on the FPGA. The quantization convolution kernel module includes:

[0148] A first LUT unit including a plurality of LUTs, used to calculate logical operations in converting a CNN convolution layer into an N×N convolution logic kernel, the N×N convolution logic kernel being converted by decomposing the CNN convolution layer into a combination of logical operations and addition operations;

[0149] A second LUT unit including a plurality of LUTs, used to calculate addition operations in converting the CNN convolution layer into the N×N convolution logic kernel;

[0150] An adder unit used to sum each branch to obtain a final result.

[0151] In the embodiment, the first LUT unit specifically calculates logical operations between weight parameters and quantized activation values of the 0th, 1st and 2nd bits in the 3×3 convolution logic kernel, that is, calculates in as well as These represent the 2nd, 1st, and 0th bits of the quantized activation value, respectively. w′1, w′2, and w′3 are the converted quantized convolution parameters for the corresponding bits. ⊙ represents the XOR sign. The second LUT unit specifically calculates the addition operation between the weight parameters in the 3×3 convolution logic kernel and the first bit of the quantized activation value. The output of the first LUT unit also has a bit concatenation circuit to concatenate the bits of each LUT62. The output of the second LUT unit also has a trailing zero padding circuit to pad the output data of the second LUT unit with zeros at the end. Figure 2 As shown, when N=3 and the quantization bits are 3, the quantization convolution kernel module specifically includes 9 LUT62 (composed of 2 LUTs), 1 LUT64 (composed of 4 LUTs), several bit operation circuits, and an adder to implement the above-mentioned 3×3 convolution logic kernel f. 3×3 (w′ i x i That is, the first LUT unit includes 9 LUT62s to perform computation. The first LUT unit has a 2-bit output with a value range of [0, 3]. Each LUT62 calculates the logical operation between the bits of the quantized activation value (bits 0, 1, and 2 of xi1 to xi9) and the weight parameters (wi1 to wi9). The second LUT unit includes one LUT64, which implements the addition function of three two-digit numbers (outmh, outmm, outml). The output is 4 bits with a value range of [0, 9], and the value range after multiplying by 2 is [0, 18]. The bit splicing circuit combines the bits of each LUT62 together, and the zero data is spliced ​​to the end of the data by the end-bit padding circuit. The adder unit implements the summation of the outputs of the four branches (out1, out2, out3, out4), that is, the four numbers out1, out2, out3, and out4 are added together to obtain the result of 3*3 convolution, and its output value range is [0, 63], which is represented by a 6-bit unsigned integer.

[0152] In this embodiment, a QCBR layer implementation module is also included, which fuses the convolutional layer with the scale layer and the activation layer. Figure 3 As shown, the QCBR layer implementation module includes:

[0153] The quantized convolution unit is used to compute each N×N convolution on the input channel using the quantized convolution kernel module to obtain the N×N convolution result.

[0154] An accumulator is used to accumulate the N×N convolution results on all input channels to obtain a convolution accumulation result; the convolution accumulation result is multiplied by the corresponding preset fusion coefficient and then added to the scale layer offset coefficient to obtain the scale operation result;

[0155] The position limiter is used for limiting the scale operation result and then outputting.

[0156] The embodiment based on the FPGA convolutional neural network quantization implementation device has the same principle as the above-mentioned FPGA convolutional neural network quantization implementation method, and will not be described one by one here.

[0157] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above with a preferred embodiment, it is not intended to limit the present application. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present application without departing from the technical solution of the present application shall fall within the scope of protection of the technical solution of the present application.

Claims

1. A method for quantization implementation of a convolutional neural network based on FPGA, characterized in that, The method comprises a quantization kernel step, comprising: The CNN convolution layer is decomposed into a combination of logical operations and addition operations, and is converted into an N*N convolution logical kernel, N being a positive integer greater than 2; The logic operation and the addition operation are respectively implemented by using a plurality of LUTs in the FPGA to implement the N*N convolution logic core; the N*N convolution logic core is obtained by combining a plurality of XNOR operations and a plurality of addition operations through bit-by-bit splitting of the quantized activation value in the CNN convolution layer according to a preset quantization bit number; the N*N convolution logic core comprises converted quantized convolution parameters The XNOR operation between each bit of the input quantized activation value is represented by using 1 bit in each operation part of the convolution logic core. The method further comprises a network training quantization step using a pre-quantization method of training and quantization, comprising: An initial CNN network N is constructed and trained to obtain a pre-training model BWN-model; Define the activation value quantization bit number q, replace the activation layer in the initial CNN network N with a preset activation layer QRelu layer to form an updated CNN network QN, and the activation value of the QRelu layer is defined as: , , is the quantization coefficient of the QRelu layer, and the initial value is ; is the input activation value of the QRelu layer, is the output activation value of the QRelu layer, and a and b are preset threshold values. The weights of each layer of the pre-training model BWN-model are loaded into the updated CNN network QN, and the updated CNN network QN is trained until the training is completed. 2.The FPGA-based convolutional neural network quantization implementation method according to claim 1, characterized in that, When N = 3, a 3x3 convolution logic kernel is implemented, multiple LUTs are used for calculation wherein are the 2nd, 1st, 0th bits of the quantized activation value respectively, w'1, w'2, w'3 are the converted quantized convolution parameters corresponding to the bits respectively, is the exclusive OR symbol, one LUT is used to implement three two-digit additions, and an adder is used to implement the summation of the outputs of each branch. 3.The FPGA-based convolutional neural network quantization implementation method according to claim 1 or 2, characterized in that, The method further comprises merging the convolution layer, the scale layer and the activation layer to form a QCBR layer, which is obtained by multiplying the N*N convolution logical kernel by a multiple coefficient and then adding a scale layer offset coefficient, so that the QCBR layer is realized using logical operations, fixed-point multiplication and fixed-point addition on the FPGA. 4.The FPGA-based convolutional neural network quantization implementation method according to claim 1, characterized in that, The steps of realizing the QCBR layer comprise: In the FPGA, the N*N convolution on each input channel is calculated using the N*N convolution logical kernel to obtain an N*N convolution result; All the N*N convolution results on the input channels are added using an adder to obtain a convolution result; The convolution addition result is multiplied by a corresponding preset multiple coefficient and then added to a scale layer offset coefficient using a multiplier-adder to obtain a scale operation result; The scale operation result is limited and then output. 5.The FPGA-based convolutional neural network quantization implementation method according to claim 1, characterized in that, In the process of training the updated CNN network QN, the quantization coefficients of each activation layer QRelu are updated in batches, and the steps comprise: Each time data from the current batch is retrieved, forward inference is performed on the current network to determine the input of each activation layer QReLU in the updated CNN network QN. If there is no number greater than 0, the quantization coefficients are updated according to the current batch; otherwise, calculation is performed. Standard deviation , The numbers greater than 0 in the QReLU of the activation layer. To quantize the mapping range, based on the standard deviation And the quantization coefficients updated in the current batch.

6. An FPGA-based convolutional neural network quantization implementation device comprising an FPGA, characterized in that, The FPGA is configured with a quantization kernel module, which comprises: A first LUT unit comprising a plurality of LUTs for calculating the logical operations in converting the CNN convolution layer into an N*N convolution logical kernel, the N*N convolution logical kernel being converted by decomposing the CNN convolution layer into a combination of logical operations and addition operations; The second LUT unit comprises a plurality of LUTs, and is configured to calculate an addition operation in the N*N convolution logic core converted from the CNN convolution layer, wherein the N*N convolution logic core is obtained by decomposing quantized activation values in the CNN convolution layer according to a preset quantization bit number, and is a combination of a plurality of XNOR operations and a plurality of addition operations; and the N*N convolution logic core comprises quantized convolution parameters after conversion The XNOR operation between each bit of the input quantized activation value is represented by 1 bit in each operation part of the convolution logic core. An adder unit for summing each LUT branch to obtain the final result; Further comprising a network training quantization module using a pre-quantization method of training and quantization, for: An initial CNN network N is constructed and trained to obtain a pre-training model BWN-model; define the activation value quantization bit number q, replace the activation layer in the initial CNN network N with a preset activation layer QRelu layer to form an updated CNN network QN, and the activation value of the QRelu layer is defined as: , , is a quantization coefficient of the QRelu layer, and the initial value is ; is an input activation value of the QRelu layer, is an output activation value of the QRelu layer, and a and b are preset threshold values. The weights of each layer of the pre-training model BWN-model are loaded into the updated CNN network QN, and the updated CNN network QN is trained until the training is completed. 7.The FPGA-based convolutional neural network quantization implementation apparatus of claim 6, wherein, The first LUT unit specifically calculates the logical operations between the weight parameters and the 0th, 1st and 2nd bits of the quantized activation value in the 3*3 convolution logical kernel, and the second LUT unit specifically calculates the addition operations of the logical operation results between the weight parameters and the 1st bit of the quantized activation value in the 3*3 convolution logical kernel, and the output end of the first LUT unit is further provided with a bit splicing circuit for bit splicing of the bit data of each LUT, and the output end of the second LUT unit is further provided with a last bit 0 supplementing circuit for last bit 0 supplementing of the output data of the second LUT unit. 8.The FPGA-based convolutional neural network quantization implementation apparatus according to claim 6 or 7, characterized in that, The QCBR layer implementation module comprises: a quantization convolution unit, configured to calculate each N×N convolution on an input channel using the quantization convolution kernel module to obtain an N×N convolution result; an accumulator, configured to accumulate the N×N convolution results on all input channels to obtain a convolution accumulation result, multiply the convolution accumulation result by a corresponding preset fusion coefficient, and add a scale layer offset coefficient to obtain a scale operation result; a limiter, configured to limit the scale operation result and output the limited result.

Images