FPGA-based convolutional neural network image processing method and device

Abstract

The application discloses a kind of based on FPGA's convolutional neural network image processing method and device, the method steps include: obtaining convolution layer configuration file, and the initialization of convolutional neural network;FPGA obtains the input of the target image to be detected to the initialized convolutional neural network, and the pre-constructed convolution logic core is used to the convolution operation of input feature map, and output feature map is obtained, the convolution logic core is converted by being decomposed into multiple logical operations and addition operation to obtain CNN convolution layer, and input feature map, output feature map are respectively stored in the two groups of specified storage space in chip during convolution calculation.The application has the advantages of simple implementation method, high processing efficiency and precision, and can realize high frame rate and low delay.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a convolutional neural network image processing method and device based on FPGA. BACKGROUND

[0002] CNN (Convolutional Neural Networks) is a kind of feedforward neural network (Feedforward Neural Networks) containing convolution calculation and having a deep structure, and is one of the representative algorithms of deep learning (deep learning). On the hardware level, the current CNN deployment is mainly based on CPU, GPU, ASIC and FPGA implementation, and each type of deployment has its own advantages and disadvantages. Compared with GPU, FPGA has the advantages of low power consumption and high speed, and also has certain cost advantage, which is suitable for embedded terminal deployment under the premise of a certain batch; and compared with ASIC, FPGA has more design space with the increasing gate resources and memory bandwidth, and FPGA also saves the flow process required in the ASIC scheme, has short development cycle and low development cost.

[0003] When using FPGA to deploy CNN, the CNN network needs to be decomposed into a structure suitable for FPGA implementation. The computing unit of FPGA is divided into DSP, multiplier-accumulator and LUT (logic lookup table), and each operation of CNN needs to be mapped to the corresponding operation logic according to the 1:1 mapping of FPGA computing unit, and then the on-chip resources are used on the FPGA side to integrate the units required for data carrying operation and data calculation operation to form a hardware operation layer.

[0004] FPGA is a field programmable gate array, which has the characteristics of high parallelism and low power consumption. When using CNN to implement image processing on FPGA, 32-bit floating-point numbers or 8-bit fixed-point numbers are usually used for convolution operation, and the input feature map and the output feature map are stored in an external memory (such as DDR). Such a scheme will have the following problems:

[0005] 1. In the case of basically consistent system performance, floating-point convolution or fixed-point convolution needs to rely on a large number of multiplications and additions, so that the implementation of CNN network quantization in FPGA will have a large amount of calculation, which cannot fully utilize the powerful logic operation capability of FPGA, resulting in the problems of large amount of calculation and low efficiency of CNN implementation, so the actual processing efficiency is not high.

[0006] 2. Current convolutional neural networks have many layers. For example, the classic neural network structure VGG16 has 16 convolutional layers and fully connected layers. When the input feature maps and output feature maps are stored in external memory in an FPGA, for VGG16, it is necessary to read the feature maps from external memory 16 times to calculate one frame of image. This makes it take a lot of time to read the input feature maps each time, which will also reduce the processing efficiency. Summary of the Invention

[0007] The technical problem to be solved by the present invention is: in view of the technical problems existing in the prior art, the present invention provides an FPGA-based convolutional neural network image processing method and device that is simple to implement, has high processing and high accuracy and efficiency, and can achieve high frame rate and low latency.

[0008] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0009] A convolutional neural network image processing method based on FPGA, comprising the following steps:

[0010] S1. Obtain the convolutional layer configuration file and weight parameters, and initialize the convolutional neural network;

[0011] S2. The FPGA acquires the target image to be detected and inputs it into the initialized convolutional neural network. It uses a pre-built convolutional logic kernel to perform convolution operation on the input feature map to obtain the output feature map. The convolutional logic kernel is obtained by decomposing the CNN convolutional layer into multiple logical operations and addition operations. During the convolution calculation process, the input feature map and the output feature map are stored in two designated on-chip storage spaces respectively.

[0012] Furthermore, in step S2, during the convolution calculation process, when calculating one layer of convolution, the input feature map is read from the first set of storage spaces within the chip, and the convolution result is written into the second set of storage spaces within the chip. When calculating the next layer of convolution, data is read from the second set of storage spaces within the chip to obtain the current input feature map, and the convolution result is written into the first set of storage spaces within the chip.

[0013] Furthermore, in step S2, the convolution calculation also includes the step of dividing the input feature map into multiple groups according to the size of the on-chip storage space.

[0014] Furthermore, in step S2, after the input feature map is input into the initialized convolutional neural network, convolution calculation, activation function and pooling calculation are performed in sequence to obtain the prediction result; the prediction result is post-processed to obtain the target recognition result, and the recognition result is transmitted through the soft core processor. The above-mentioned input of the input feature map, calculation of the convolutional neural network and post-processing steps are executed in parallel.

[0015] Further, in step S2, a plurality of LUTs are used in the FPGA to respectively implement the logical operation and the addition operation, so as to implement the N*N convolution logic core, which is obtained by decomposing the quantized activation value in the CNN convolution layer according to a preset quantization bit number, and combining a plurality of exclusive OR operations and a plurality of addition operations.

[0016] Further, in step S1, the training server is used to train the convolution layer configuration file and the weight parameter in the convolution neural network, and the training server is used to transmit the convolution layer configuration file and the weight parameter to the soft core processor and store the convolution layer configuration file and the weight parameter in the DDR. The FPGA obtains the convolution layer configuration file and the weight parameter required for initialization from the DDR, and the recognition result output by the FPGA is transmitted to the soft core processor through the DDR.

[0017] A FPGA-based convolution neural network image processing device, comprising a FPGA, wherein the FPGA is configured with:

[0018] An input module, configured to obtain a convolution layer configuration file and a weight parameter to initialize a convolution neural network, and obtain a target image to be detected;

[0019] A CNN calculation module, configured to input the target image to be detected into the initialized convolution neural network, and use a pre-constructed N*N convolution logic core to implement N*N convolution operation on an input feature map, so as to obtain an output feature map, wherein the N*N convolution logic core is converted by decomposing a CNN convolution layer into a plurality of logical operations and addition operations;

[0020] Two groups of on-chip storage spaces, configured to respectively store the input feature map and the output feature map in the convolution calculation process.

[0021] Further, the device further comprises a training server, a soft core processor and a DDR connected in sequence, another end of the DDR is connected with the FPGA, the training server is configured to train a convolution layer configuration file and a weight parameter in a convolution neural network, and transmit the convolution layer configuration file and the weight parameter to the soft core processor and store the convolution layer configuration file and the weight parameter in the DDR, and the output result of the CNN calculation module is transmitted to the soft core processor through the DDR.

[0022] Further, the CNN calculation module comprises a quantized convolution kernel unit for implementing the N*N convolution logic core, and the quantized convolution kernel unit comprises:

[0023] A first LUT unit comprising a plurality of LUTs, configured to calculate logical operations for converting a CNN convolution layer into an N*N convolution logic core, wherein the N*N convolution logic core is converted by decomposing the CNN convolution layer into a combination of logical operations and addition operations;

[0024] a second LUT unit comprising a plurality of LUTs for calculating addition operations in the N*N convolution logic core converted from the CNN convolution layer;

[0025] an adder unit for summing the data output by the first LUT unit and the second LUT unit to obtain a final result.

[0026] Further, the first LUT unit comprises 9 LUTs 62, each of which is composed of 2 LUTs 6, and the LUT 6 is the smallest programmable unit of the FPGA, the second LUT unit comprises 1 LUT 64, which is composed of 4 LUTs 6, and the output end of the first LUT unit is further provided with a bit splicing circuit, and the output end of the second LUT unit is further provided with a last bit 0 supplementing circuit, each bit of data of each LUT 62 is combined together by the bit splicing circuit, and 0 data is spliced to the end of the data by the last bit 0 supplementing circuit.

[0027] Compared with the prior art, the application has the following advantages:

[0028] 1. The application converts the CNN convolution layer into a convolution logic core containing only logical operations and addition operations by decomposing the CNN convolution layer into logical operations and addition operations, and uses the convolution logic core in the FPGA to realize convolution operation on the input feature map, so that the powerful logical operation capability of the FPGA can be fully utilized, and the input feature map and the output feature map are respectively stored in two groups of storage spaces in the chip during the convolution calculation process, so that the number of times of reading the input feature map from outside can be greatly reduced, the processing efficiency can be effectively improved, and high frame rate and low delay image processing can be realized.

[0029] 2. The application further supports storage of the input feature map and the output feature map by the on-chip storage space by partitioning the input feature map, and can be applied to processing of input feature maps of various sizes and channel numbers, so that the deployment of a complex convolutional neural network on the FPGA can be facilitated under the premise of ensuring the recognition accuracy of the deep learning neural network.

[0030] 3. The application further uses INT3 fixed-pointing of the input and output feature maps and INT1 fixed-pointing of the weight, which can effectively reduce the use of FPGA resources, complete more convolution calculations in less resources, enable more convolution kernels to be deployed in the case of limited resources, and effectively improve the frame rate, enhance the fluency and real-time performance of processing. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a schematic diagram of the implementation process of the FPGA-based convolutional neural network image processing method of the embodiment.

[0032] Figure 2It is a structure principle schematic diagram of the embodiment based on FPGA to realize convolutional neural network image processing.

[0033] Figure 3 It is a flow schematic diagram of the embodiment based on CNN to realize target detection.

[0034] Figure 4 It is a structure principle schematic diagram of the embodiment (3*3 convolution) based on FPGA to realize convolution calculation. DETAILED DESCRIPTION

[0035] The application is further described below in combination with the drawings of the specification and specific preferred embodiments, but the protection scope of the application is not limited by this.

[0036] The network model quantization of CNN mainly includes two parts, one is the quantization of weight, and the other is the quantization of activation value. When the weight and activation value are quantized to 8 bits, the performance equivalent to 32 bits can be obtained. The basic operation in the neural network is the convolution and multiplication-addition operation of the weight and activation value. If one of them is quantized to {-1, 1}, the multiplication-addition operation can be simplified to addition-subtraction operation. If both of them are quantized to {-1, 1}, the multiplication-addition operation can be simplified to bitwise operation. Through the above simplification of CNN quantization to addition-subtraction and bitwise operation, the quantization process can be easily realized by hardware. Meanwhile, in the convolution process, if the input feature map and the output feature map are directly stored in the on-chip memory, the input feature map can be directly read in the convolution process without frequent reading operation, and a large amount of reading time can be reduced.

[0037] Based on the above consideration, the CNN convolution layer is decomposed into logical operation and addition operation, and is converted into a convolution logic core containing only logical operation and addition operation. The convolution logic core is used to realize convolution operation on the input feature map in FPGA. Since only logical operation and addition operation are needed, the powerful logical operation capability of FPGA can be fully utilized. Meanwhile, the input feature map and the output feature map are respectively stored in two groups of storage spaces in the chip during the convolution calculation process, so that the number of reading input feature map from outside can be greatly reduced, the processing efficiency can be effectively improved, and high frame rate and low delay image processing can be realized.

[0038] As shown in Figure 1 The steps of the convolutional neural network image processing method based on FPGA of the embodiment include:

[0039] S1. The FPGA acquires the convolution layer configuration file and the weight parameter, and initializes the convolutional neural network.

[0040] S2. The FPGA obtains the target image to be detected and inputs the same to the initialized convolutional neural network, uses a pre-constructed convolution logic core to realize convolution operation on the input feature map, obtains an output feature map, and the convolution logic core is converted by decomposing the CNN convolution layer into multiple logical operations and addition operations. The input feature map and the output feature map are respectively stored in two groups of designated storage spaces in the chip during the convolution calculation.

[0041] As shown in Figure 2 In step S1 of the embodiment, the convolution layer configuration file and the weight parameter in the convolutional neural network are trained using a training server, transmitted to a soft core processor, and stored in a DDR. The FPGA obtains the convolution layer configuration file and the weight parameter required for initialization from the DDR. The convolution layer configuration file and the weight parameter are updated online by the soft core processor. When new convolution layer configuration file and weight parameter are obtained by training, the FPGA obtains the new convolution layer configuration file and weight parameter from the DDR for initialization. If there is no new convolution layer configuration file and weight parameter, the historical convolution layer configuration file and weight parameter stored in the DDR are used. The recognition result output by the FPGA is transmitted to the soft core processor through the DDR, and the recognition result is sent to a peripheral device by the soft core processor for further decision processing. The soft core processor can be a CPU or a FPGA soft core. The CNN model and the training parameter can be updated remotely by the soft core processor.

[0042] In step S2 of the embodiment, during the convolution calculation, the input feature map is read from the first group of storage spaces in the chip when a layer of convolution is calculated, and the convolution result is written into the second group of storage spaces in the chip. When the next layer of convolution is calculated, the data is read from the second group of storage spaces in the chip to obtain the current input feature map, and the convolution result is written into the first group of storage spaces in the chip. Therefore, the input feature map is directly read from the storage space in the FPGA chip each time the convolution calculation is performed, without the need to read from the external memory, which can greatly reduce the number of times of reading the input feature map and improve the efficiency of the convolution calculation. Under the premise of ensuring the recognition accuracy of the deep learning neural network, the deployment of the complex convolutional neural network on the FPGA can be facilitated.

[0043] The on-chip storage space can be URAM or BLOCK RAM, that is, two groups of URAM / BLOCK RAM are pre-divided in the chip, the input feature map and the output feature map are stored in the two groups of URAM / BLOCK RAM, when the first layer of convolution is calculated, the first group of URAM / BLOCK RAM is read to obtain the input feature map, the convolution result is written into the second group of URAM / BLOCK RAM, when the second layer of convolution is calculated, the second group of URAM / BLOCK RAM is read to obtain the input feature map, the convolution result is written into the first group of URAM / BLOCK RAM, and the polling is performed in turn. It can be understood that the division and type of the on-chip storage space can be selected according to actual needs in addition to the above manner.

[0044] The internal storage space (such as URAM / BLOCK RAM) resource in the FPGA is limited, and the size of the input feature map can be large or the number of channels is large, so it is difficult to directly store the input feature map into the internal storage space (URAM / BLOCK RAM) of the FPGA. In step S2 of the embodiment, the input feature map is divided into multiple groups according to the size of the on-chip storage space during convolution calculation, that is, the input feature map is divided into multiple groups, for example, when the size of the input feature map is 32*512*1024, the input feature map can be divided into 32*(512 / N)*1024 N equal parts according to the size of the URAM / BLOCK RAM, when N=4, that is, 32*128*1024. By dividing the input feature map, the input feature map and the output feature map can be stored in the on-chip storage space, and the input feature map of various sizes and channel numbers can be processed.

[0045] In step S2 of the embodiment, after the input feature map is input into the initialized convolutional neural network, convolution calculation, activation function and pooling calculation are performed in turn to obtain a prediction result; the prediction result is post-processed to obtain a target recognition result, and the recognition result is transmitted through the soft core processor. For example, Figure 2 , 3As shown, in the present embodiment, when image target recognition is performed, after the image to be detected is acquired and preprocessed such as image scaling, feature extraction, boundary box regression and object class prediction are completed by using the convolutional neural network, the features are extracted from the backbone CNN network, so as to obtain the boundary box position and class prediction data, and the convolutional neural network is sequentially subjected to convolution layer, activation layer and pooling layer for calculation, the full connection layer is used to combine the channel results to obtain the final feature extraction result, the feature extraction result is used for prediction, and the input feature map and the output feature map in the above calculation process are respectively stored in the two groups of URAM / BLOCK RAM in the chip; then, the prediction information obtained by the convolutional neural network is post-processed to obtain the target recognition result, the target recognition result is transmitted to the CPU / FPGA soft core processor through the DDR, the CPU / FPGA soft core processor is further sent to the peripheral device for further decision, and the detection result can be further displayed on the original image.

[0046] The convolutional neural network has a feedforward hierarchical structure, data is transmitted from front to back in the network layer by layer, so there is data dependency between different network layers, the parallelism that can be developed is very small, and the standard convolution operation in the same convolution layer is a multi-dimensional operation, which can be summarized as three levels of input feature map, input feature map and convolution window. In step S2 of the present embodiment, the input of the input feature map, the calculation of the convolutional neural network and the post-processing steps are executed in parallel, which can further improve the calculation speed and ensure the processing efficiency.

[0047] As shown in Figure 2 , 3As shown, to realize the above method, a set of devices is formed by a training server, a CPU / FPGA soft core, a DDR, an FPGA, a camera, and peripheral equipment in the specific application embodiment of the application. The training server is used for deep learning training server, mainly for parameter tuning, pruning, floating point to fixed point, and identification accuracy evaluation. The CPU / FPGA soft core loads the convolution parameter and the convolution layer configuration file first. The FPGA receives the video stream from the camera and performs calculation, and writes the final CNN result into the DDR. The CPU / FPGA soft core provides the final identification result to the peripheral equipment. The CPU / FPGA soft core can be an arm, microblaze, etc. processor, mainly used to update the new CNN convolution layer configuration file and weight parameter trained by the training server, and send the CNN identification result to the peripheral equipment. The peripheral equipment is a decision device, etc. For example, in license plate access control identification, whether it is the vehicle in the park is judged according to the identification result, so as to make the action of access control. D is used for interaction between the CPU / FPGA soft core and the FPGA, and is used to store the convolution layer configuration file, the weight parameter, and the identification result. The main function of the FPGA is to receive the camera video stream, video stream scaling, convolution calculation, activation function, pooling, and post-processing. The camera is used to collect graphics in real time as the input of the whole system.

[0048] In the specific application embodiment, the detailed steps of image processing by using the above device and the above method of the application are as follows:

[0049] Step 1: System initialization

[0050] After the system is powered on, the CPU / FPGA soft core communicates with the training server. If the training server has new training weight parameters or convolution layer configuration files, the new weight and convolution layer configuration file are transmitted to the CPU / FPGA soft core through the network port and written into the DDR. If the training server has no new training weight parameters or convolution layer configuration files, the stored historical weight parameters and convolution layer configuration files are written into the DDR.

[0051] Step 2: CNN calculation

[0052] After the FPGA writes the convolution layer configuration file into the BRAM / URAM, the video stream from the camera is received, and the graphics scaling is executed in the form of flow and written into the BRAM / URAM. The convolution calculation, activation function, and pooling calculation are executed and written into the BRAM / URAM. The post-processing calculation is performed, and the calculation result is written into the DDR. The above steps are parallel processing to improve the calculation speed.

[0053] Step 3: Result sending

[0054] The post-processing result is read from the DDR by the CPU / FPGA soft core, and the final classification result is transmitted to the peripheral device through a network cable.

[0055] In step S2 of this embodiment, a plurality of LUTs are used in the FPGA to respectively implement logic operations and addition operations, so as to implement an N*N convolution logic core. The N*N convolution logic core is obtained by splitting the quantized activation value in the CNN convolution layer according to a preset quantization bit number, and decomposing the quantized activation value into a combination of a plurality of XNOR operations and a plurality of addition operations.

[0056] Taking N=3 and a quantization bit number of 3 as an example, the process of converting the quantized convolution formula to obtain a 3*3 convolution logic core is as follows:

[0057]

[0058] In the above formula, w is a quantized convolution parameter, the value of which is [-1, 1], and c is the number of input channels; w' is a converted quantized convolution parameter, the value of which is [0, 1], and the value of w is mapped to each other; x is a 3-bit quantized activation value, the value of which is [0, 1, …, 6, 7], x 2 is the second bit of the quantized activation value, the value of which is [0, 1]; x 1 is the first bit of the quantized activation value, the value of which is [0, 1]; x 0 is the zeroth bit of the quantized activation value, the value of which is [0, 1].

[0059] In the above formula, the quantized activation value x ij is first split by bits, that is, Since the value of w i,j is [-1, 1], the value of w' is [0, 1], and after multiplication, there will be 3-value cases, which cannot be represented by 1 bit, so w' is converted to w After conversion, the result of multiplication is still [-1, 1] which is 2-value; since the value of w is [-1, 1], 1 bit cannot represent it, so w is replaced by w through the formula w Since w i,j and c are known numbers, that is, β c is a constant term. The above 3*3 convolution logic core is obtained based on 3-bit quantization bit decomposition, and each part only needs to be represented by 1 bit. In the FPGA, the above 1bW3bA 3*3 convolution kernel logic can be implemented by LTU (logic lookup table) and an adder.

[0060] To implement the above 3*3 convolution logic core f 3×3 (w' ix i ), specifically, a plurality of LUTs can be used to calculate wherein and are the 2nd, 1st, 0th bits of the quantized activation value, w'1, w'2, w'3 are the converted quantized convolution parameters corresponding to the bits, one LUT is used to implement the addition of three two-digit numbers, and an adder is used to implement the summation of the outputs of each branch.

[0061] LUT6 is the smallest programmable unit of FPGA, an arbitrary 6-input Boolean expression, and the operation result is one bit of data. In a specific application embodiment, LUT6 is used to implement 3-bit feature Figure 1 bit weight 3x3 convolution process as Figure 4 shown, using 9 LUT62s (composed of 2 LUT6s), 1 LUT64 (composed of 4 LUT6s), a plurality of bit operation circuits and an adder to implement the above 3x3 convolution logic core f 3×3 (w′ i ,x i ), LUT62 is used to calculate function, i.e., logical operation of each bit of the input feature map and the weight parameter, LUT64 implements the addition function of three two-digit numbers, a1 to a9 in the figure are the lowest bits of 9 3-bit input feature maps, b1 to b9 are the middle bits of 9 3-bit input feature maps, c1 to c9 are the highest bits of 9 3-bit input feature maps, w1 to w9 are 9 1-bit weights input. LUT62 is 2 LUT6s, the inputs of the two LUTs are exactly the same, only the truth table is different, LUT64 is 4 LUT6s, the bit splicing circuit is arranged at the output end of each LUT62, which is used to combine each bit of data together and does not occupy LUT resources; the last bit 0 filling circuit outputs at the output end of LUT64, which is used to splice 1 bit 0 data to the end of the data, also does not occupy FPGA resources, finally X1, X2, X3, X4 are added up to obtain the result of 3x3 convolution.

[0062] The embodiment specifically adopts INT3 for convolution calculation in the main network, the first layer convolution and classification regression convolution adopt INT16 calculation, and INT3 represents eight numerical values of 0, 1, 2, 3, 4, 5, 6 and 7 without a decimal part, that is, 3*3 convolution of 1bit weight is performed on 3bit feature map, the characteristics of FPGA can be fully utilized, only 37 LUTs are needed, and no DSP resource is occupied, the resource usage can be effectively reduced while the accuracy is ensured. If the feature map is A1 to A9 of traditional INT8, and the weight is INT8 of W1 to W9, 3*3 convolution is realized as A1*W1+A2*W2+...A9*W9, 9 DSPs and 65 LUT6s are needed. In the embodiment, the input and output feature maps are INT3 fixed-point, and the weight is INT1 fixed-point, so that the FPGA resource usage can be effectively reduced, more convolution calculations can be completed in less resources, more convolution kernels can be deployed in the case of limited resources, the frame rate can be effectively improved, and the fluency and real-time performance of processing can be enhanced.

[0063] In addition to the above-mentioned quantization of 3 bits, other quantization bit numbers (quantization bit numbers greater than or equal to 2) can of course be used, that is, an n-bit activation value is represented as: x i represents the ith bit of x; then the same reasoning is used according to the above-mentioned quantization convolution formula, and different FPGA implementation modes can be obtained when different quantization bit numbers are used. For example, when the quantization bit number n = 4, the 3*3 convolution logic core f 3×3 (w′ i ,x i ) is In addition to the above-mentioned 3*3 convolution logic core, other N*N convolution can of course be constructed, and the principle is the same as above.

[0064] In the embodiment, the CNN convolution layer is decomposed into multiple logical operations and multiple addition operations according to the quantization bit n, and an N*N convolution logic core is constructed, each operation part in the convolution logic core can be represented by 1 bit, so that the N*N convolution logic core can be efficiently realized by LTU and adder in FPGA. By decomposing the CNN network into a structure suitable for FPGA implementation, the calculation amount, memory capacity and read-write bandwidth required by the CNN can be greatly reduced.

[0065] The embodiment is specifically realized by using the verilog language, compared with the traditional realization by using xilinx HLS or Altrea Open CL, the controllability can be effectively improved, and the resource utilization rate can be improved.

[0066] The FPGA-based convolutional neural network image processing device in the embodiment includes an FPGA, and the FPGA is configured with:

[0067] an input module configured to obtain a convolutional layer configuration file and weight parameters to initialize a convolutional neural network, and obtain a target image to be detected;

[0068] a CNN calculation module configured to input the target image to be detected into the initialized convolutional neural network, and perform convolution operation on an input feature map using a pre-constructed convolution logic kernel to obtain an output feature map, the convolution logic kernel being converted from the CNN convolutional layer by decomposing the CNN convolutional layer into a plurality of logical operations and addition operations;

[0069] two groups of on-chip memory spaces configured to store the input feature map and the output feature map respectively during the convolution calculation.

[0070] In a specific application embodiment, the FPGA-based convolutional neural network image processing device as shown in FIG. 1 includes an input module, a CNN calculation module, and two groups of on-chip memory spaces. Figure 2 The input module includes an image scaling pre-processing unit, the CNN calculation module includes a convolution, activation function, pooling, and full connection layer calculation unit, and the two groups of on-chip memory spaces store the input feature map and the output feature map.

[0071] As shown in FIG. 2, the convolutional neural network image processing device further includes a training server, a soft core processor, and a DDR connected in sequence. Figure 2 The training server is configured to train the convolutional layer configuration file and the weight parameters in the convolutional neural network, transmit the same to the soft core processor, and store the same in the DDR.

[0072] In this embodiment, the CNN calculation module includes a quantized convolution kernel unit for implementing an N*N convolution logic kernel.

[0073] The first LUT unit includes a plurality of LUTs, and is configured to calculate logical operations for converting the CNN convolutional layer into the N*N convolution logic kernel.

[0074] The second LUT unit includes a plurality of LUTs, and is configured to calculate addition operations for converting the CNN convolutional layer into the N*N convolution logic kernel.

[0075] The adder unit is configured to sum each branch to obtain a final result.

[0076] In a specific application embodiment, the quantized convolution kernel unit as shown in FIG. 3 includes a first LUT unit, a second LUT unit, and an adder unit. Figure 4As shown, the first LUT unit includes 9 LUTs 62, each of which is composed of 2 LUTs 6, the smallest programmable unit of FPGA, and the second LUT unit includes 1 LUT 64, which is composed of 4 LUTs 6. The output end of the first LUT unit is further provided with a bit splicing circuit, and the output end of the second LUT unit is further provided with a last-bit 0 supplementing circuit. Each bit data of each LUT 62 is combined together by the bit splicing circuit, and 0 data is spliced to the end of the data by the last-bit 0 supplementing circuit, that is, the above-mentioned 3x3 convolution logic core is realized by using 9 LUTs 62, 1 LUT 64, a plurality of bit operation circuits and an adder 3×3 (w′ i ,x i Each LUT 62 calculates the logical operation between each bit (a1 to a9, b1 to b9 and c1 to c9) of the input feature map and the weight parameter (w1 to w9), each bit data is combined together by the bit splicing circuit, 1 bit 0 data is spliced to the end of the data by the last-bit 0 supplementing circuit, and finally X1, X2, X3 and X4 are added to obtain the result of 3x3 convolution.

[0077] The convolutional neural network image processing device of the embodiment is consistent with the principle of the above-mentioned convolutional neural network image processing method, and will not be described one by one here.

[0078] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above with a preferred embodiment, it is not intended to limit the present application. Therefore, any simple modification, equivalent change and modification made to the above embodiment without departing from the technical solution of the present application, according to the technical essence of the present application, should fall within the scope of protection of the technical solution of the present application.

Claims

1. A method for FPGA-based convolutional neural network image processing, characterized by the steps of Comprise: S1. Obtain the convolution layer configuration file and weight parameters, and initialize the convolutional neural network; S2. The FPGA obtains the target image to be detected and inputs it into the initialized convolutional neural network, and uses the pre-constructed convolution logic core to realize convolution operation on the input feature map, to obtain the output feature map, the convolution logic core is converted by decomposing the CNN convolution layer into multiple logic operations and addition operations, and the input feature map and the output feature map are respectively stored in two groups of specified storage spaces in the chip during the convolution calculation process; In step S2, multiple LUTs are used in the FPGA to implement the logical operations and addition operations respectively, thereby realizing an N*N convolutional logic kernel. The N*N convolutional logic kernel is decomposed into a combination of multiple XOR operations and multiple addition operations by bitwise splitting the quantized activation values ​​in the CNN convolutional layer according to a preset quantization bit depth. Each operation part in the convolutional logic kernel is represented by 1 bit. The logical operations and addition operations in the N*N convolutional logic kernel are implemented by multiple LUTs in the FPGA respectively, thereby realizing the N*N convolutional logic kernel. In the implementation of the N*N convolutional logic kernel, when N=3, the calculation is performed by LUT62 in the first LUT unit. The function, namely the logical operation between the input feature map bits and the weight parameters, is implemented by LUT64 in the second LUT unit, which performs three two-digit additions to sum the data output from the first and second LUT units to obtain the final result. LUT62 consists of two LUT6 units, and LUT64 consists of four LUT6 units. The OR symbol, These are the 2nd, 1st, and 0th bits of the quantized activation value, respectively. These are the quantized convolution parameters for the corresponding bits. 2.The FPGA-based convolutional neural network image processing method of claim 1, wherein, In the step S2, during the convolution calculation process, the input feature map is read out from the first group of storage spaces in the chip, and the convolution result is written into the second group of storage spaces in the chip, and the next layer of convolution is calculated, the data is read from the second group of storage spaces in the chip to obtain the current input feature map, and the convolution result is written into the first group of storage spaces in the chip. 3.The FPGA-based convolutional neural network image processing method of claim 1, wherein, In the step S2, the input feature map is divided into multiple groups according to the size of the storage space in the chip during the convolution calculation. 4.The FPGA-based convolutional neural network image processing method of claim 1, wherein, In the step S2, after the input feature map is input into the initialized convolutional neural network, convolution calculation, activation function and pooling calculation are sequentially performed to obtain a prediction result; the prediction result is post-processed to obtain a target recognition result, and the recognition result is transmitted through the soft core processor, and the input of the input feature map, the calculation of the convolutional neural network and the post-processing of each step are executed in parallel. 5.The FPGA-based convolutional neural network image processing method according to any one of claims 1-4, characterized in that, In the step S1, the training server is used to train the convolution layer configuration file and weight parameters in the convolutional neural network, which are transmitted to the soft core processor and stored in the DDR, and the FPGA obtains the convolution layer configuration file and weight parameters required for initialization from the DDR, and the recognition result output by the FPGA is transmitted to the soft core processor through the DDR.

6. An FPGA-based convolutional neural network image processing apparatus comprising an FPGA, characterized in that, The FPGA is configured with: An input module for obtaining the convolution layer configuration file and weight parameters to initialize the convolutional neural network, and obtaining the target image to be detected; A CNN calculation module for inputting the target image to be detected into the initialized convolutional neural network, using the pre-constructed N*N convolution logic core to realize N*N convolution operation on the input feature map, and obtaining the output feature map, the N*N convolution logic core is converted by decomposing the CNN convolution layer into multiple logic operations and addition operations; Two groups of on-chip storage spaces for respectively storing the input feature map and the output feature map during the convolution calculation process; In the CNN calculation module, a plurality of LUTs are used in the FPGA to respectively implement the logical operation and the addition operation, to implement the N*N convolution logic core. The N*N convolution logic core is decomposed into a combination of a plurality of XNOR operations and a plurality of addition operations by bit-by-bit splitting of the quantized activation value in the CNN convolution layer according to a preset quantization bit number. Each operation part in the convolution logic core is represented by 1 bit. The logical operation and the addition operation in the N*N convolution logic core are respectively implemented by a plurality of LUTs in the FPGA, to implement the N*N convolution logic core. When N=3, the calculation is implemented by LUT62 in the first LUT unit The function, i.e. the logical operation of each bit of the input feature map and the weight parameter, is implemented by LUT64 in the second LUT unit. The first LUT unit and the second LUT unit output data are summed to obtain the final result. LUT62 is composed of 2 LUT6s, and LUT64 is composed of 4 LUT6s, The XNOR symbol, The second, first and zero bits of the quantized activation value, The converted quantized convolution parameters corresponding to the bits. 7.The FPGA-based convolutional neural network image processing apparatus of claim 6, wherein, Further comprising a training server, a soft core processor and a DDR connected in sequence, the other end of the DDR is connected with the FPGA, the training server is used to train the convolution layer configuration file and weight parameters in the convolutional neural network, which are transmitted to the soft core processor and stored in the DDR, and the output result of the CNN calculation module is transmitted to the soft core processor through the DDR.

8. The FPGA-based convolutional neural network image processing apparatus according to claim 6 or 7, characterized in that, The CNN calculation module comprises a quantization convolution kernel unit for realizing the N*N convolution logic core, and the quantization convolution kernel unit comprises: The first LUT unit includes a plurality of LUTs for calculating logical operations in the N*N convolution logic core converted from the CNN convolution layer; The second LUT unit includes a plurality of LUTs for calculating addition operations in the N*N convolution logic core converted from the CNN convolution layer; The adder unit is used for summing the data output by the first LUT unit and the second LUT unit to obtain a final result. 9.The FPGA-based convolutional neural network image processing apparatus of claim 8, wherein, The first LUT unit includes 9 LUTs 62, each of which is composed of 2 LUTs 6, and the LUT 6 is the minimum programmable unit of the FPGA. The second LUT unit includes 1 LUT 64, which is composed of 4 LUTs 6. The output end of the first LUT unit is further provided with a bit splicing circuit, and the output end of the second LUT unit is further provided with a last bit 0 supplementing circuit. The bit data of each LUT 62 is combined together by the bit splicing circuit, and the 0 data is spliced to the end of the data by the last bit 0 supplementing circuit.

Images