A point cloud target detection neural network accelerator and acceleration method based on FPGA

Abstract

The application belongs to the field of FPGA-based hardware accelerator design, and proposes a point cloud target detection neural network accelerator and an acceleration method based on FPGA. The convolution accelerator comprises an instruction decoding module, a convolution module, an accumulation module, a truncation module and an activation function. The intermediate value cache only temporarily stores the channels of one feature map, consumes less BRAM, and is suitable for accelerating the point cloud target detection neural network with large feature map size, large number of channels and large network size. The application has flexible configurability and excellent reusability, so the application can accelerate different convolution layers of the point cloud target detection neural network, and is suitable for being integrated as an IP into a large design, reducing FPGA resource consumption and improving resource utilization. Moreover, the weight and the input feature map share one data input channel, and the convolution intermediate value is temporarily stored on-chip BRAM, reducing the feature map carrying frequency and saving the FPGA interface bandwidth.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of FPGA-based hardware accelerator design, in particular to a point cloud target detection neural network accelerator and an acceleration method based on FPGA. BACKGROUND

[0002] Since the point cloud target detection neural network SA-SSD, VoxelNet, PointPillars, etc. includes a large number of convolution layers, the computing power consumed by the network is also mainly used for convolution calculation, and the convolution operation is composed of a large number of multiplication and accumulation operations, so the point cloud target detection neural network needs to be deployed to a high-performance computing platform. Common computing platforms include CPU, GPU, FPGA, ASIC, etc. Compared with other computing platforms, FPGA has the advantages of high parallelism, large computing power, flexible design scheme modification, low power consumption, etc., and the verified FPGA scheme can also provide a basis for developing ASIC, so it has certain practical significance to develop a point cloud target detection neural network accelerator based on FPGA.

[0003] The existing non-reusable accelerator, such as 'Li Li, Chen Xinyu, Gao Wenbin. An Implementation Scheme of Convolutional Neural Network Accelerator Based on FPGA [J]. Journal of Beijing University of Electronic Science and Technology, 2022, 30(04): 96-104' and 'He Kaiquan, Yuan Xun, Chen Song. Hardware Architecture Design of Convolutional Neural Network Based on FPGA Dynamic Reconfiguration [J]. Information Technology and Network Security, 2019, 38(03): 77-81', is built and optimized for a specific convolutional neural network, and the parameters of each layer, such as the size of the convolution kernel, the size of the input feature map, the step, and the padding, are different and fixed. The reusability of the accelerator is poor. Since the current layer of the neural network needs to wait for the previous layer to finish computing before starting to compute, it will cause high resource occupancy and low utilization of FPGA. The existing reusable accelerator, such as 'He Jiajun, Su Chengyue, Luo Rongfang, et al. Research and Design of Quantized Inference CNN Acceleration System Based on FPGA [J]. Computer Measurement and Control, 2022, 30(09): 162-169', improves the reusability, but the frequent transfer of feature maps reduces the computing efficiency of FPGA. The convolutional layer of the point cloud target detection neural network is different from the convolutional neural network. The point cloud has sparsity, and the feature map extracted from the point cloud usually has larger size and more channels. The point cloud target detection neural network usually has a large number of layers. If the existing non-reusable convolution accelerator and acceleration method is used for the point cloud target detection neural network, the Flip-Flop (FF), Look-Up-Table (LUT), and BlockRAM (BRAM) consumed by each layer of the accelerator will greatly exceed the resource limit of FPGA, and the waiting time between adjacent convolution layers will be longer, causing serious resource idling and waste. Therefore, a point cloud target detection neural network accelerator with excellent reusability, low resource consumption, and high performance is designed based on FPGA. Through multi-layer reuse, configuration of different parameters, and reduction of feature map transfer times, the point cloud target detection network can efficiently utilize FPGA resources and computing power, which has important practical significance in the field of point cloud target detection. SUMMARY

[0004] To solve the above problems, the present application provides a point cloud target detection neural network accelerator and acceleration method based on FPGA. By using quantized weights and input feature maps, real-time configuration of convolution parameters of each module, and increasing the parallelism of computation, the same convolution accelerator can adapt to different layers of the point cloud target detection network, reuse the convolution accelerator, reduce the resource occupancy of FPGA, and improve the resource utilization and computing performance.

[0005] The technical scheme of the present application is as follows:

[0006] The application discloses an FPGA-based point cloud target detection neural network accelerator, which comprises an instruction decoding module, a convolution module, an accumulation module, a truncation module and an activation function; control signals are input into the convolution module, the accumulation module and the truncation module through the instruction decoding module; and a data stream is sequentially output through the convolution module, the accumulation module, the truncation module and the activation function.

[0007] The instruction decoding module is used for decoding control signals carried on an instruction bus, and obtaining parameters required by the convolution module, the accumulation module and the truncation module.

[0008] The convolution module is used for buffering quantized weight and input feature map data, and performing parallel convolution operation; the convolution module comprises a convolution control module, a weight buffer, an input buffer, a multiplier array and an addition tree; the convolution control module is used for controlling parallel convolution calculation progress, and controlling data reading of the weight buffer and the input buffer; the convolution control module is internally provided with a weight counter w_cnt, a row counter row_cnt, a column counter col_cnt, a step row counter s_row_cnt and a step column counter s_col_cnt; the weight buffer is used for buffering weight data, and outputting the weight data to the multiplier array; the input buffer mainly comprises a shift register and an input register array, and is used for buffering input feature map data by rows, and outputting the input feature map data to the multiplier array; the weight buffer and the input buffer share a data input channel, and different data are sequentially input into the data input channel; the multiplier array is composed of a plurality of multipliers, and is used for performing a plurality of multiplication operations in parallel convolution operation; the addition tree is composed of a plurality of adders, each adder is composed of a plurality of two-input one-output adders, and each adder adds a plurality of input data two by two in one clock cycle, and outputs a result to a next adder; the addition tree is used for summing a plurality of multiplication results output by the multiplier array in a plurality of clock cycles, and outputting a summation result as an intermediate value to the accumulation module;

[0009] The accumulation module is used for storing different batches of intermediate values output by the convolution module, and accumulating the different batches of intermediate values corresponding to pixels; after accumulation is completed, an output feature map is generated, and then a bias is added to the output feature map, and the output feature map is output to the truncation module; the accumulation module comprises an accumulation and control module and an intermediate value buffer; the accumulation and control module is used for controlling accumulation progress, reading a summation result of a previous batch from the intermediate value buffer, adding the read summation result of the previous batch and a current batch intermediate value output by the addition tree to obtain a summation result of the current batch, and writing the summation result of the current batch into the intermediate value buffer; after accumulation of all batches of intermediate values is completed, an output feature map is obtained, and then a bias adding operation is performed on the output feature map; the intermediate value buffer is mainly composed of BRAMs, and is used for buffering a first batch of intermediate values calculated by the convolution module and buffering a summation result of a current batch calculated by the accumulation and control module; the accumulation and control module is internally provided with an accumulation batch counter parl_cnt;

[0010] a truncation module for truncating the bit width of the biased output feature map, including one multiplier;

[0011] an activation function for processing the truncated output feature map through the activation function and outputting.

[0012] The decoding process of the instruction decoding module is as follows: the instruction bus instr is sequentially truncated according to the parameter bit width to obtain the convolution accelerator identifier id, the convolution accelerator enable en, the input feature map width w, the input feature map height h, the input channel c, the step s, the padding p, the input feature map discard width w_dsc, the input feature map discard height h_dsc, the truncation bit length m_shift, the truncation coefficient m_caps, and the bias b; wherein w_dsc=(w+2×p-F)%s, h_dsc=(h+2×p-F)%s, F is the convolution kernel size of the convolution layer of the point cloud target detection neural network corresponding to the convolution accelerator, and "%" represents the remainder operation; m_shift and m_caps are two integer statistics used to reduce truncation error in the quantization process, and the specific operation is to multiply the output feature map data generated by the accumulation module by m_caps, and the product is right shifted by m_shift bits.

[0013] A point cloud target detection neural network acceleration method based on FPGA, including the following steps:

[0014] Step one, convolution module calculation;

[0015] Step 1.1: The input data channel is first connected to the weight buffer, and the weight buffer loads the weight data with a parallel degree of PARL;

[0016] Step 1.2: The convolution control module records the number of weights loaded into the weight buffer through its built-in weight counter w_cnt, and when w_cnt=F×F, the input data channel is connected to the input buffer, and step 1.3 is executed, and the bit width of the weight data loaded into the weight buffer is quantized to BW;

[0017] Step 1.3: The input buffer loads the input feature map data with a parallelism of PARL, and uses F-1 shift registers with a depth of w+2×p-3 to buffer F-1 rows of input feature map, and the column counter in the convolution control module is incremented by 1 synchronously with the loading of each pixel data; the input buffer includes an input register array with a size of F×F, and the output values of the F-1 shift registers are stored in the F-1 rows of the Fth column of the input register array, the input feature map data loaded by the input buffer is stored in the Fth column of the Fth row of the input register array, and the value of the i+1th column of the input register array is stored in the ith column, 1≤i<F, at this time, the F×F values of the input register array are the F×F data required by the multiplier array, so as to form a F×F convolution window in the input feature map, and the convolution window slides with the loading of the input feature map data one by one; the bit width of the input feature map data loaded by the quantized input buffer is consistent with the bit width of the weight data loaded by the quantized weight buffer, and both are BW;

[0018] Step 1.4: The convolution control module records the position of the input buffer loading the input feature map data through the row counter row_cnt and the column counter col_cnt, that is, the row number and the column number of the current F×F convolution window in the input feature map, when 0≤col_cnt≤w+2×p-1 and 0≤row_cnt≤h+2×p-h_dsc-1, it indicates that the convolution window is in the convolution effective area;

[0019] Step 1.5: The convolution control module controls the convolution effectiveness through the step row counter s_row_cnt and the step column counter s_col_cnt, the step row counter s_row_cnt is incremented synchronously with the row counter row_cnt, and when col_cnt≤w+2×p-w_dsc-1, the step column counter s_col_cnt is incremented synchronously with the column counter col_cnt; when the step row counter s_row_cnt reaches the upper limit s-1, the step row counter s_row_cnt is reset to 0, and when the step column counter s_col_cnt reaches the upper limit s-1, the step column counter s_col_cnt is reset to 0;

[0020] Step 1.6: The area col_cnt<p, col_cnt>w+p-1, row_cnth+p-1 of the input feature map is a padding area, when the convolution window is in the padding area, the convolution control module sends a control signal to disconnect the input data channel, and loads 0 into the input buffer; when the convolution window is in the non-padding area, the convolution control module sends a control signal to open the input data channel and receives the input feature map data;

[0021] Step 1.7: When the step row counter s_row_cnt and the step column counter s_col_cnt are both 0, the convolution control module sends a convolution valid signal, and the convolution module starts the convolution operation, specifically, the weight buffer sends FxF parallel degree PARL weights to the multiplier array, the input buffer sends FxF parallel degree PARL input feature map data contained in the input register array to the multiplier array, and the weight with a bit width of BW and the input feature map data with a bit width of BW are subjected to multiplication operation in the multiplier array with a size of PARLxFxF, to generate PARLxFxF results with a bit width of 2xBW-1, which are output to the addition tree;

[0022] Step 1.8: The addition tree is formed by cascading N stages of adders; for the first stage of adders, when the number of output data M1 of the multiplier array is even, the input data of the addition tree are added two by two, and M1 / 2 results are output; when M1 is odd, the first M1-1 input data of the addition tree are added two by two, M1-1 / 2 sums are obtained, and the M1th input data of the addition tree is taken as the output together; for the jth stage of adders, when the number of output data Mj of the previous stage of adders is even, the output data of the previous stage of adders are added two by two, and Mj / 2 results are output; when Mj is odd, the first Mj-1 output data of the previous stage of adders are added two by two, Mj-1 / 2 sums are obtained, and the Mjth output data of the previous stage of adders is taken as the output together; wherein the number of stages N of the adders is determined by 2^(N-1)<PARLxFxF≤2^N when the convolution accelerator is deployed, the bit width of the output result of each of the first N-1 stages of adders is increased by 1 compared to the bit width of the output result of the previous stage, and the upper limit of the bit width of the output result of each stage of adders is 32; the bit width of the output result of the first stage of adders is 2xBW, the bit width of the output result of the N-1th stage of adders is 2xBW+N-2, and the bit width of the output result of the Nth stage of adders is 32; wherein 1<j≤N, and "^" represents exponentiation;

[0023] Step 2, the accumulation module calculates;

[0024] Step 2.1: The accumulation and control module calculates the maximum accumulation batch value parlmax=c / PARL, when parlmax=1, no accumulation, and the accumulation and control module directly outputs the intermediate value output by the convolution module to the truncation module after adding the bias b; when parlmax>1, step 2.2 is executed for accumulation; wherein the bit width of the intermediate value output by the convolution module is 32, and the bit width of the bias b is 2xBW, and the upper limit is 32;

[0025] Step 2.2: The accumulation and control module records the batch of the intermediate value output by the convolution module using an accumulation batch counter parl_cnt, which is incremented when the rlast signal in the input data channel is pulled high, and parl_cnt is reset to 0 when parl_cnt = parlmax-1 and the rlast signal is pulled high, where the rlast signal represents the last pixel of the current batch of intermediate values output by the convolution module;

[0026] Step 2.3: The accumulation and control module controls the accumulation progress using a state machine with S0, S1, S2, S3 four states, S0 is an idle state, when the rvalid signal is valid and parl_cnt = 0, the state machine jumps to S1 state, the rvalid signal represents that the current batch of intermediate values output by the convolution module is valid;

[0027] Step 2.4: In S1 state, the accumulation and control module writes the current batch of intermediate values output by the convolution module to the intermediate value cache, and jumps to S2 state when the rlast signal is valid, and parl_cnt is incremented to 1; wherein the data bit width of the intermediate value cache is 32 bits;

[0028] Step 2.5: In S2 state, when parl_cnt = 2, the accumulation and control module reads the last batch of intermediate values from the intermediate value cache and adds them to the current batch of intermediate values output by the convolution module to obtain the current batch of summation results; when parl_cnt > 2, the accumulation and control module reads the last batch of summation results from the intermediate value cache and adds them to the current batch of intermediate values output by the convolution module to obtain the current batch of summation results; the current batch of summation results obtained by addition is written to the intermediate value cache, completing the accumulation and update of the intermediate value, and when rlast is valid and parl_cnt = parlmax-2, jump to S3 state, and parl_cnt is incremented to parlmax-1;

[0029] Step 2.6: In S3 state, the accumulation and control module reads the last batch of summation results from the intermediate value cache and adds them to the current batch of intermediate values output by the convolution module to obtain the current batch of summation results; jump to S0 state when the rlast signal is valid, and parl_cnt is reset to 0;

[0030] Step 2.7: The accumulation and control module adds the current batch of summation results in S3 state in step 2.6 to the bias b to complete the bias addition operation, and outputs the result of the bias addition operation to the truncation module; the output result of the accumulation module has a bit width of 32;

[0031] Step three, the truncation module calculates;

[0032] Step 3.1: Calculate the upper limit of the truncation for a bit width of BW, clipmax = 2^(BW-1)-1, then the truncation interval is [-clipmax, clipmax];

[0033] Step 3.2: Multiply the acc_res output of the accumulator module (32 bits wide) with m_caps through the multiplier of the truncated module to obtain the product res (32 bits wide). Shift res right by m_shift bits to obtain res_shift (32 bits wide).

[0034] Step 3.3: When res_shift > clipmax, the truncation result exceeds the upper limit of the truncation range, and clipmax is output to the activation function; when res_shift < -clipmax, the truncation result exceeds the lower limit of the truncation range, and -clipmax is output to the activation function; when -clipmax ≤ res_shift ≤ clipmax, the truncation result belongs to the truncation range, then the bits from BW-1 to 0 of res_shift are taken as res_clip and output to the activation function; where the bit width of res_clip is BW.

[0035] Step 4: Activation function calculation; The input activation function data is compared with 0 using the comparator principle. If the input activation function data is greater than 0, it is output directly; if the input activation function data is less than 0, 0 is output. The bit width of the activation function output result is BW.

[0036] The beneficial effects of this invention are:

[0037] 1. The intermediate value cache of this invention only temporarily stores one channel of the feature map, consumes less BRAM, and is suitable for accelerating point cloud target detection neural networks with large feature map size, many channels, and large network size.

[0038] 2. This invention features flexible configurability and excellent reusability. Configurability includes specifying data bit width, convolutional kernel size, and parallelism by passing parameters before deploying the accelerator, and changing parameters such as input feature map size, input channels, and stride in real time by sending instructions after deployment. Reusability includes changing instructions to accelerate different convolutional layers, and instantiating multiple instances of the invention to increase the parallelism of convolution acceleration. Therefore, this invention can accelerate different convolutional layers of point cloud object detection neural networks and is suitable for integration as IP into large-scale designs, reducing FPGA resource consumption and improving resource utilization.

[0039] 3. In this invention, the weights and the input feature map share a single data input channel, and the intermediate values ​​of the convolution are temporarily stored in the on-chip BRAM, which reduces the number of feature map transfers and saves FPGA interface bandwidth. Attached Figure Description

[0040] Figure 1 is a FPGA-based point cloud target detection neural network accelerator architecture diagram of the application; Figure 1 In the figure, the thick arrow is the data flow, and the thin arrow is the control signal.

[0041] Figure 2 is an instruction bus structure diagram, where the numerical subscript is the relative position of each instruction.

[0042] Figure 3 is a multiplication array and addition tree calculation flowchart. Figure 3 In the figure, the arrow points to the serial number of the input data of this stage, and the block represents only the register operation.

[0043] Figure 4 is an accumulation module state transition flowchart.

[0044] Figure 5 is a truncation module truncation flowchart. DETAILED DESCRIPTION

[0045] In order to make the technical solutions of the application clearer, the application will be further described below in conjunction with the drawings and examples.

[0046] As shown in Figure 1 , the FPGA-based point cloud target detection neural network accelerator described in this embodiment comprises an instruction decoding module, a convolution module, an accumulation module, a truncation module and an activation function, the convolution module comprises a convolution control module, a weight cache, an input cache, a multiplier array and an addition tree, and the accumulation module comprises an accumulation and control module and an intermediate value cache.

[0047] This embodiment selects a convolution layer CONV in the Backbone of Pointpillars as the acceleration object, and details the acceleration process of the convolution accelerator proposed by the application to the point cloud target detection network, and the parameters of CONV are shown in Table 1.

[0048] Table 1 Convolution layer structure parameters

[0049] Convolutional layer Input size Convolution kernel size Stride Padding Output size CONV 64×248×216 3×3 1 1 64×248×216

[0050] The specific structure and parameters of the convolution accelerator configured in this embodiment are:

[0051] The deployment parameters of the convolution accelerator are: BW=8, F=3, PARL=16, the parallelism of the quantized weight and input feature map is 16, each pixel contains 16 channels of data, and the data bit width of each channel is 8, so the bit width of each weight and input feature map pixel is 16x8=128; the activation function is ReLU.

[0052] The instruction decoding module is as shown in Figure 2As shown, the instruction bus bit width is 75, the contained parameters, bit width and values are shown in Table 2, wherein the value of the bias b changes with the current output feature map channel;

[0053] Table 2 Instruction bus parameter table

[0054]

[0055]

[0056] Convolution module parameters: the convolution effective area in the convolution control module is 0≤col_cnt≤w+2×p-1=217, 0≤row_cnt≤h+2×p-h_dsc-1=249, the upper limit of the step row and column counters s_row_cnt and s_col_cnt is s-1=0, the value is always 0, the padding area is col_cnt<p=1 or col_cnt>w+p-1=216 or row_cnt<p=1 or row_cnt>h+p-1=248; the number of weights in the weight cache is F×F=9; the number of shift registers in the input cache is F-1=2, the shift register depth is w+2×p-3=215; the number of multipliers in the multiplier array is F×F×PARL=144; the number of adders in the addition tree is N=8;

[0057] Accumulation module parameters: the maximum value of the accumulation batch in the accumulation and control module is parlmax=c / PARL=4; after the rvalid signal is valid, the accumulation and control module jumps to the S1 state, parl_cnt=0, the accumulation and control module is in the S1 state, 0<parl_cnt≤parlmax-2=2, the accumulation and control module is in the S2 state, parl_cnt=parlmax-1=3, the accumulation and control module is in the S3 state;

[0058] Clipping module parameters: the upper limit of clipping is clipmax=2^(BW-1)-1=127, then the clipping interval is [-clipmax, clipmax]=[-127, 127].

[0059] The specific calculation process of the embodiment is as follows:

[0060] The convolution module calculation steps are:

[0061] Step 1.1: The quantized weight is loaded into the weight cache through the data input channel, and when the weight counter in the convolution control module is 9, the weight cache stops loading;

[0062] Step 1.2: the quantized input feature map is loaded into the input buffer via the data input channel, and the row counter and the column counter in the convolution control module are counted synchronously; the input buffer uses two shift registers to store two rows of input feature map, and the output values of the two shift registers are stored in the third column of the first two rows of the input register array, the input feature map data is stored in the third column of the third row of the input register array, the values in the third column of each row of the input register array are stored in the second column of each row, and the values in the second column are stored in the first column; at this time, the 9 values of the input register array are the 9 data required by the multiplier array, which forms a 3x3 convolution window in the input feature map, and the convolution window slides with the loading of the input feature map data one by one;

[0063] Step 1.3: the row counter and the column counter record the position of the convolution window, when the convolution window is in the padding area, the input buffer will temporarily stop receiving external data, and 0 will be loaded as input data;

[0064] Step 1.4: the step row counter and the step column counter in the convolution control module are always 0, when the convolution window is in the padding area or the convolution valid area, the convolution starts, the weight buffer inputs 9 weights into the multiplier array, and the input buffer inputs 9 input feature map data into the multiplier array, the multiplier array performs 144 multiplication operations and inputs the products into the addition tree; wherein the bit width of each input data of the multiplier array is 8, and the bit width of the output data is 32;

[0065] Step 1.5: there are 8 levels of adders in the addition tree, as shown in Figure 3 , the input data number of the first to fourth level adders is even, so each level adds two inputs, and 9 sums are generated after the 144 products pass through the first to fourth level adders; the input data number of the fifth to seventh level adders is odd, so each level adds two inputs, and the remaining one input is passed to the next level, and 2 sums are generated after the 9 sums pass through the fifth to seventh level adders, and finally one sum is generated after passing through the eighth level adder, and the sum is output to the accumulation module; wherein the input and output data bit width of the adder is 32;

[0066] The accumulation module is shown in Figure 4 , and the calculation steps are:

[0067] Step 2.1: after each pull-up of the last pixel indication signal rlast, the accumulation batch counter parl_cnt in the accumulation and control module is incremented by 1, until 4;

[0068] Step 2.2: after the input data valid signal rvalid is pulled up, the state of the accumulation and control module jumps from S0 to S1;

[0069] Step 2.3: parl_cnt = 0 in S1 state, the accumulation and control module stores the received data into the intermediate value buffer, and after rlast is pulled high, the state of the accumulation and control module jumps from S1 to S2;

[0070] Step 2.4: 0 < parl_cnt ≤ 2 in S2 state, the accumulation and control module first takes out the data stored in the previous batch from the corresponding position of the intermediate value buffer, then adds the taken-out data and the received data, and then stores the sum in the corresponding position of the intermediate value buffer, so that the data in the intermediate value buffer is accumulated and updated, parl_cnt = 2 and after rlast is pulled high, the state of the accumulation and control module jumps from S2 to S3;

[0071] Step 2.5: parl_cnt = 3 in S3 state, the accumulation and control module first takes out the data stored in the previous batch from the corresponding position of the intermediate value buffer, then adds the taken-out data and the received data, and after rlast is pulled high, the state of the accumulation and control module jumps from S3 to S0, and parl_cnt will be set to 0;

[0072] Step 2.6: the accumulation and control module adds the sum in S3 state in step 2.5 and the bias b, and outputs the result to the truncation module; wherein the input and output data of the accumulation and control module and the data in the intermediate value buffer have a bit width of 32, and the bias b has a bit width of 16;

[0073] The calculation step of the truncation module is:

[0074] Step 3.1: the data acc_res output by the accumulation module is multiplied by m_caps = 99 through a multiplier, to obtain a product res with a bit width of 32, and res is right-shifted by m_shift = 14 to obtain res_shift with a bit width of 32, as shown in Figure 5 , wherein the high 14 bits of res_shift are extended sign bits, and the low 18 bits are the high 18 bits of res;

[0075] Step 3.2: if res_shift is in the truncation interval [-127, 127], the low 8 bits of res_shift are taken as the result and output to the activation function; if res_shift > 127, 127 is output to the activation function; if res_shift < -127, -127 is output to the activation function;

[0076] The calculation step of the activation function is: if the data output by the truncation module is less than 0, the activation function outputs 0, otherwise the activation function outputs the data itself; wherein the input and output data of the activation function have a bit width of 8.

[0077] The accelerator of the embodiment is deployed on a Pynq-Z2 development board of Xilinx Company, and the FPGA resource consumption is shown in Table 3.

[0078] Table 3 FPGA resource table

[0079]

[0080] The total operation number of the convolution layer CONV is 3935748096, the clock period is 13715712, and the throughput of the accelerator is about 57.39GOPS at a clock frequency of 200MHz, and the time is about 68.58ms. The performance comparison of the accelerator of the embodiment is shown in Table 4, wherein the comparison accelerator 1 selects the accelerator described in “He J, Su C, Luo R, et al. Research and design of quantized inference CNN acceleration system based on FPGA[J]. Computer Measurement & Control, 2022, 30(09): 162-169.”, and the comparison accelerator 2 selects the accelerator described in “Zhang K, Zhao S, Sun Q, et al. Design of multi-core scalable convolution accelerator based on FPGA[J]. Computer Engineering and Design, 2021, 42(06): 1592-1598.”. The DSP efficiency of the FPGA of the embodiment is higher than that of other accelerators, so the accelerator consumes less resources and has high resource utilization.

[0081] Table 4 Performance comparison

[0082]

Claims

1. An FPGA-based point cloud object detection neural network accelerator, characterized in that, The FPGA-based point cloud target detection neural network accelerator comprises an instruction decoding module, a convolution module, an accumulation module, a truncation module and an activation function; control signals are input to the convolution module, the accumulation module and the truncation module through the instruction decoding module; The data flow is sequentially output through the convolution module, the accumulation module, the truncation module and the activation function; The instruction decoding module is used for decoding the control signals carried on the instruction bus, and obtaining the parameters required by the convolution module, the accumulation module and the truncation module; The convolution module is used for buffering the quantized weight and input feature map data, and performing parallel convolution operation; the convolution module comprises a convolution control module, a weight buffer, an input buffer, a multiplier array and an addition tree; the convolution control module is used for controlling the progress of parallel convolution calculation, and controlling the data reading of the weight buffer and the input buffer; the convolution control module is internally provided with a weight counter w_cnt, a row counter row_cnt, a column counter col_cnt, a step row counter s_row_cnt and a step column counter s_col_cnt; the weight buffer is used for buffering weight data, and outputting the weight data to the multiplier array; the input buffer mainly comprises a shift register and an input register array, and is used for buffering the input feature map data by rows, and outputting the input feature map data to the multiplier array; the weight buffer and the input buffer share a data input channel, and different data are sequentially input; the multiplier array is composed of a plurality of multipliers, and is used for performing a plurality of multiplication operations in parallel convolution operation; the addition tree is composed of a plurality of adders, each adder is composed of a plurality of two-input one-output adders, and each adder adds a plurality of input data two by two in one clock cycle, and outputs the result to the next adder; the addition tree is used for summing a plurality of multiplication results output by the multiplier array in a plurality of clock cycles, and outputting the summation result as an intermediate value to the accumulation module; The accumulation module is used for storing different batches of intermediate values output by the convolution module, and accumulating the different batches of intermediate values corresponding to pixels; after the accumulation is completed, an output feature map is generated, and then a bias is added to the output feature map and output to the truncation module; the accumulation module comprises an accumulation and control module and an intermediate value buffer; The accumulation and control module is used for controlling the accumulation progress, reading the summation result of the previous batch from the intermediate value buffer, adding the read summation result of the previous batch to the current batch intermediate value output by the addition tree to obtain the summation result of the current batch, and writing the summation result of the current batch into the intermediate value buffer; after the accumulation of all batches of intermediate values is completed, an output feature map is obtained, and then a bias adding operation is performed on the output feature map; the intermediate value buffer is mainly composed of BRAMs, and is used for buffering the first batch of intermediate values calculated by the convolution module and buffering the summation result of the current batch calculated by the accumulation and control module; the accumulation and control module is internally provided with an accumulation batch counter parl_cnt; The truncation module is used for truncating the bit width of the output feature map after bias addition, and comprises one multiplier; The activation function is used for outputting the truncated output feature map after processing by the activation function.

2. The FPGA-based point cloud object detection neural network accelerator of claim 1, wherein, The decoding process of the instruction decoding module is: the instruction bus instr is sequentially intercepted according to the parameter bit width to obtain the convolution accelerator identifier id, the convolution accelerator enable en, the input feature map width w, the input feature map height h, the input channel c, the step s, the padding p, the input feature map discard width w_dsc, the input feature map discard height h_dsc, the shift length m_shift, the shift coefficient m_caps and the bias b; wherein w_dsc=(w+2×p-F)%s, h_dsc=(h+2×p-F)%s, F is the convolution kernel size of the convolution layer of the point cloud target detection neural network corresponding to the convolution accelerator, and "%" represents the remainder operation; m_shift and m_caps are two integer statistics used to reduce the shift error in the quantization process, and the specific operation is to multiply the output feature map data generated by the accumulation module by m_caps, and the product is right shifted by m_shift.

3. An FPGA-based point cloud target detection neural network acceleration method, characterized in that, The steps include the following: Step 1, the convolution module calculates; Step 1.1: the input data channel is first connected to the weight buffer, and the weight buffer loads the weight data with a parallelism of PARL; Step 1.2: the convolution control module records the number of weights loaded into the weight buffer through the built-in weight counter w_cnt, when w_cnt=FxF, the input data channel is connected to the input buffer, and step 1.3 is executed, and the bit width of the weight data loaded into the weight buffer is BW; Step 1.3: the input buffer loads the input feature map data with a parallelism of PARL, and uses F-1 cascaded shift registers with a depth of w+2×p-3 to cache F-1 rows of input feature map, and the column counter in the convolution control module is incremented by 1 synchronously every time one pixel data is loaded; the input buffer includes an input register array with a size of FxF, the output values of the F-1 shift registers are respectively stored in the Fth column of the first F-1 rows of the input register array, the input feature map data loaded by the input buffer is stored in the Fth column of the Fth row of the input register array, and the value of the i+1th column of the input register array is stored in the ith column, 1≤iF, at this time, the FxF values of the input register array are the FxF data required by the multiplier array, so as to form a FxF convolution window in the input feature map, and the convolution window slides with the loading of the input feature map data; the bit width of the input feature map data loaded into the input buffer is consistent with the bit width of the weight data loaded into the weight buffer, which is BW; Step 1.4: the convolution control module records the position of the input feature map data loaded by the input buffer through the row counter row_cnt and the column counter col_cnt, that is, the row number and the column number of the current FxF convolution window in the input feature map, when 0≤col_cnt≤w+2×p-1, 0≤row_cnt≤h+2×p-h_dsc-1, it indicates that the convolution window is in the convolution effective area; Step 1.5: The convolution control module controls the convolution validity through a step row counter s_row_cnt and a step column counter s_col_cnt, the step row counter s_row_cnt is incremented synchronously with the row counter row_cnt, and the step column counter s_col_cnt is incremented synchronously with the column counter col_cnt when col_cnt≤w+2×p-w_dsc-1; when the step row counter s_row_cnt reaches the upper limit s-1, the step row counter s_row_cnt is reset to 0, and when the step column counter s_col_cnt reaches the upper limit s-1, the step column counter s_col_cnt is reset to 0; Step 1.6: The region col_cnt<p, col_cnt>w+p-1, row_cnth+p-1 of the input feature map is a padding region, when the convolution window is in the padding region, the convolution control module sends a control signal to disconnect the input data channel, and loads 0 into the input buffer; Step 1.7: When the step row counter s_row_cnt and the step column counter s_col_cnt are both 0, the convolution control module sends a convolution valid signal, and the convolution module starts the convolution operation, specifically, the weight buffer sends F×F parallel degree PARL weights to the multiplier array, the input buffer sends F×F parallel degree PARL input feature map data contained in the input register array to the multiplier array, the weight with a bit width of BW and the input feature map data with a bit width of BW are subjected to multiplication operation in the multiplier array with a size of PARL×F×F, and PARL×F×F results with a bit width of 2×BW-1 are output to the addition tree. Step 1.8: The addition tree is formed by cascading N stages of adders; for the first stage of adders, when the number of output data of the multiplier array M1 = PARL x F x F is even, the input data of the addition tree are added two by two, and M1 / 2 results are output; when M1 is odd, the first M1-1 input data of the addition tree are added two by two, (M1-1) / 2 sums are obtained, and the M1th input data of the addition tree is taken together as the output; for the jth stage of adders, when the number of output data of the previous stage of adders Mj is even, the output data of the previous stage of adders are added two by two, and Mj / 2 results are output; when Mj is odd, the first Mj-1 output data of the previous stage of adders are added two by two, (Mj-1) / 2 sums are obtained, and the Mjth output data of the previous stage of adders is taken together as the output; wherein the number of stages of adders N is determined by 2^(N-1) < PARL x F x F ≤ 2^N when the convolution accelerator is deployed, the bit width of the output result of each of the first N-1 stages of adders is increased by 1 compared to the bit width of the output result of the previous stage, and the bit width of the output result of each stage of adders is up to 32; the bit width of the output result of the first stage of adders is 2 x BW, the bit width of the output result of the N-1th stage of adders is 2 x BW + N-2, and the bit width of the output result of the Nth stage of adders is 32; wherein 1 < j ≤ N, and "^" represents exponentiation; Step 2, the accumulation module calculates; Step 2.1: The accumulation and control module calculates the maximum accumulation batch parlmax = c / PARL, when parlmax = 1, no accumulation, the accumulation and control module directly outputs the intermediate value output by the convolution module to the truncation module after adding the bias b; when parlmax > 1, step 2.2 is performed for accumulation; wherein the bit width of the intermediate value output by the convolution module is 32, and the bit width of the bias b is 2 x BW, up to 32; Step 2.2: The accumulation and control module uses the accumulation batch counter parl_cnt to record the batch of the intermediate value output by the convolution module, parl_cnt increases with the rising of the rlast signal in the input data channel, when parl_cnt = parlmax-1 and the rlast signal is pulled high, parl_cnt is reset to 0, wherein the rlast signal represents the last pixel of the current batch of intermediate values output by the convolution module; Step 2.3: The accumulation and control module uses a state machine with S0, S1, S2, S3 four states to control the accumulation progress, S0 is the idle state, when the rvalid signal is valid and parl_cnt = 0, the state machine jumps to S1 state, the rvalid signal indicates that the current batch of intermediate values output by the convolution module is valid; Step 2.4: In S1 state, the accumulation and control module writes the current batch of intermediate values output by the convolution module into the intermediate value cache, and jumps to S2 state when the rlast signal is valid, and parl_cnt is incremented to 1; wherein the data bit width of the intermediate value cache is 32 bits; Step 2.5: In S2 state, when parl_cnt = 2, the accumulation and control module reads the last batch intermediate value from the intermediate value cache and adds it to the current batch intermediate value output by the convolution module to obtain the current batch summation result; when parl_cnt > 2, the accumulation and control module reads the last batch summation result from the intermediate value cache and adds it to the current batch intermediate value output by the convolution module to obtain the current batch summation result; The current batch summation result obtained by addition is written into the intermediate value cache, completing the accumulation and update of the intermediate value, and when rlast is valid and parl_cnt = parlmax-2, jumping to S3 state, parl_cnt is incremented to parlmax-1; Step 2.6: In S3 state, the accumulation and control module reads the last batch summation result from the intermediate value cache and adds it to the current batch intermediate value output by the convolution module to obtain the current batch summation result; when the rlast signal is valid, jumping to S0 state, and resetting parl_cnt to 0; Step 2.7: The accumulation and control module adds the current batch summation result in S3 state in step 2.6 to the bias b to complete the bias addition operation, and outputs the result of the bias addition to the clipping module; the output result of the accumulation module has a bit width of 32; Step 3, the clipping module calculates; Step 3.1: Calculate the clipping upper limit clipmax = 2^(BW-1)-1 with a bit width of BW, and the clipping interval is [-clipmax, clipmax]; Step 3.2: Multiply the output acc_res of the accumulation module with a bit width of 32 and m_caps through the multiplier of the clipping module to obtain the product res with a bit width of 32, and right shift res by m_shift bits to obtain res_shift with a bit width of 32; Step 3.3: When res_shift > clipmax, the clipping result exceeds the clipping upper limit, and clipmax is output to the activation function; when res_shift < -clipmax, the clipping result exceeds the clipping lower limit, and -clipmax is output to the activation function; when -clipmax ≤ res_shift ≤ clipmax, the clipping result belongs to the clipping interval, then the BW-1th to 0th bits of res_shift are truncated as res_clip, and output to the activation function; wherein res_clip has a bit width of BW; Step 4, the activation function calculates; the data input to the activation function is compared with 0 through the comparator principle, when the data input to the activation function is greater than 0, it is directly output; when the data input to the activation function is less than 0, 0 is output; wherein the bit width of the output result of the activation function is BW.

Images