Hardware Acceleration Implementation of TB-Net for DoA Estimation

Abstract

The present invention belongs to the field of wireless communication technology, and specifically is a TB‑Net hardware acceleration implementation method for DoA estimation. The method of the present invention comprises the following steps: designing the architecture design of the TB‑Net accelerator for DoA estimation; designing the data flow, and writing a script that can reorder the weights and biases of TB‑Net; designing a custom simple instruction set, and writing TB‑Net into an instruction program; quantizing the weights and bias data of TB‑Net into 16-bits signed integer data; performing circuit function verification, and comparing software and hardware results. The circuit structure includes: an instruction decoding module, an instruction storage module, a global data cache module, a data transmission network, a PE array, and a global control module. The present invention uses fixed weights and pulse arrays to increase the number of data reuses, significantly reduce the amount of data transmission, and can adapt to one-dimensional convolutions of various sizes, so as to realize a low-power and high-speed hardware accelerator for DoA using TB‑Net.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and wireless communication technology, and provides a TB-Net hardware acceleration implementation method for DoA estimation. Background Art

[0002] In recent decades, DoA (Direction of Arrival) estimation has been widely studied in the fields of acoustics, radar, sonar, and wireless communications. Traditional DoA estimation methods, such as Multiple Signal Classification (MUSIC) and Signal Parameter Estimation Based on Rotation Invariant Technique (ESPIRIT) rely on accurate signal models, and their DoA estimation accuracy will be significantly reduced when the model is imperfect.

[0003] In recent years, with the rapid development of deep learning, DoA estimation algorithms based on neural networks have been proposed. These methods are generally divided into two types: regression networks and classification networks. Due to the data-driven characteristics, these methods are robust to model defects. In the article "TB-NET: A Two-Branch Neural Network for Directionof Arrival Estimation under Model Imperfections", Lin Liyu et al. from Fudan University combined the classification network and regression network structures widely used in DoA estimation by current deep learning, used the two structures in parallel, and designed a two-branch neural network that combines classification and regression in parallel, denoted as TB-Net (Two-Branch Neural Network), and optimized the classification network branch to improve the accuracy of the classification network branch's rough estimation of DoA. This TB-Net ensures that the system is more robust to model defects while reducing the amount of calculation by making the network lightweight.

[0004] TB-Net used for DoA still has a certain computational complexity. If the CPU is used alone for calculation, it will take a long time to complete, which cannot meet the high real-time requirements of DoA. Therefore, it is necessary to use hardware such as GPU, ASIC or FPGA to accelerate the calculation, so as to realize the real-time reasoning of the neural network. Although GPU has powerful parallel computing capabilities, the power consumption of GPU usually reaches about 100W, which seriously limits the application scenarios. ASIC has the advantages of high energy efficiency and small area, but the development cycle is relatively long and the cost is high. Although the hardware performance of FPGA is not as good as ASIC, the development cycle is shorter and the cost is lower. FPGA and ASIC can be selected according to the actual situation to realize the neural network accelerator. Summary of the invention

[0005] The purpose of the present invention is to provide a TB-Net hardware acceleration implementation method for DoA estimation, so as to improve the hardware energy efficiency of TB-Net, reduce the calculation delay, and realize real-time DoA estimation.

[0006] The TB-Net (Two-Branch Neural Network) hardware acceleration implementation method for DoA (Direction of Arrival) estimation provided by the present invention comprises the following specific steps:

[0007] Step 1: Design the architecture of the TB-Net accelerator for DoA estimation:

[0008] Step 2: Design the data flow and write a script that can reorder the weights and biases of TB-Net;

[0009] Step 3: Design a custom simple instruction set and write TB-Net into an instruction program;

[0010] Step 4: Quantize the weight and bias data of TB-Net into 16-bit signed integer data;

[0011] Step 5: Verify circuit function and compare software and hardware results.

[0012] The specific steps of the accelerator architecture design in step 1 are as follows:

[0013] Step 1-1: Design the data path from the host to the accelerator, store the weight and bias data into DRAM through the PCI-E data path, and read the calculation results from the accelerator;

[0014] Step 1-2: Design the convolution calculation module PE array; suppose the PE array has 8*8 PE units, a single PE unit is used to process a single-channel convolution calculation of a convolution kernel, each column of PE corresponds to the convolution calculation of different channels of the same convolution kernel, and each row of PE corresponds to the convolution calculation of the same channel of different convolution kernels; then the entire PE array can perform 8-channel convolution operations of 8 convolution kernels at the same time, and the calculation results of each column of PE are added to obtain the convolution result of eight channels; for the convolution layer calculation with the number of convolution kernels and the number of channels less than 8, the output eigenvalues ​​are added with the bias and stored in the eigenvalue cache; for the convolution layer calculation with the number of convolution kernels or the number of channels greater than 8, batch processing is required; wherein the number of convolution kernels and the number of channels of each batch shall not exceed 8; then batch processing is performed in the same way as above; for example, for the convolution calculation with 16 convolution kernels and 16 channels, it needs to be divided into four blocks M 0-7 C 0-7 ,M 0-7 C 8-15 ,M8-15 C 0-7 ,M 8-15 C 8-15 ; Among them, M 0-7 C 0-7 represents the weight blocks of channels 0 to 7 of the 0th to 7th convolution kernels, and the rest are represented in the same way as above; in the first batch of channels (M 0-7 C 0-7 ,M 8-15 C 0-7 ), the convolution result is added with the bias to obtain the output eigenvalue, and it is stored in the eigenvalue cache; for the non-first batch of channels (M 0-7 C 8-15 ,M 8-15 C 8-15 ), add the obtained convolution result to the convolution result of the previous channel of the same convolution kernel in the eigenvalue cache to obtain the output eigenvalue, and store it in the eigenvalue cache;

[0015] Step 1-3: Design a single PE unit in step 1-2; a single PE contains a weight storage module with an address depth of 8, 8 multipliers and 8 accumulators, and supports one-dimensional convolution calculations with convolution kernel lengths of 1 to 8; first load the corresponding weight data in DRAM into the PE to fix the weight, and then load a feature value data from the feature value cache into the PE in each clock cycle. The feature value is multiplied with the weight value fixed by the PE at the same time, and accumulated with the product of the previous clock cycle, and the output feature value of the corresponding convolution calculation is output at the corresponding time;

[0016] Step 1-4: Design an on-chip cache to temporarily store input eigenvalues, bias data and intermediate results; for the storage of intermediate results, the eigenvalue cache will be equally divided into two parts - eigenvalue cache 1 and eigenvalue cache 2; for a layer of calculation of the convolutional network, eigenvalue cache 1 is used to store the input eigenvalues ​​of the convolution calculation of this layer, and eigenvalue cache 2 is used to store the output eigenvalues ​​of the convolution calculation of this layer; for the next layer of convolution calculation, the output eigenvalues ​​of the previous layer stored in eigenvalue cache 2 are the input eigenvalues ​​of the convolution calculation of this layer, and eigenvalue cache 1 is used to store the output eigenvalues; if there is a next layer of convolution calculation, the functional conversion between eigenvalue cache 1 and eigenvalue cache 2 will be similar; eigenvalue cache 1 and eigenvalue cache 2 are equally divided into 8 parts respectively to store eigenvalues ​​of different channels; the data output end of each cache corresponds to the input eigenvalue end of a row of PE arrays in step 1-2, and the data input end of each cache corresponds to the output eigenvalue end of a row of PE arrays in step 1-2;

[0017] Step 1-5: Design the control module, including the clock generator for controlling the convolution step size, the convolution padding generator, the flag signal generator during data transmission and calculation, and a series of address pointer generators.

[0018] Design the data flow described in step 2 and write a script that can reorder the weights and biases of TB-Net. The steps are to store the weights and biases in DRAM according to the convolution calculation order of the PE array. When performing convolution calculation, the accelerator only needs to read data from DRAM according to the sequential pointer; the convolution calculation order of the PE array is:

[0019] Step 2-1: Read C×M×W×H weight data from DRAM and store them in the PE array; C is the number of convolution kernel channels, M is the number of convolution kernels, W is the width of the convolution kernel, and H is the height of the convolution kernel; the convolution kernel width of TB-Net used for DoA estimation is 1, so it can be simplified to C×M×H weight data;

[0020] Step 2-2: Read C×W input eigenvalues ​​from the eigenvalue cache in each clock cycle; C is the number of input eigenvalue channels, which is the same as the number of convolution kernel channels, and W is the input eigenvalue width; the input eigenvalue width of TB-Net used for DoA estimation is 1, so it can be simplified to read C input eigenvalues ​​in each clock cycle; these C input eigenvalues ​​are multiplied by the C×M×H weights stored in the PE array to obtain C×M×H intermediate results; for a single PE, one input eigenvalue is multiplied by the H weights stored in it to obtain H intermediate results;

[0021] Step 2-3: In each clock cycle, the H intermediate results obtained in a single PE unit are staggered and accumulated with the intermediate results obtained in the previous cycle. After H clock cycles, the convolution result of the first convolution slice mapped on the eigenvalue can be obtained, and the convolution result of the second convolution slice can be obtained in the next clock cycle; and so on, if the input eigenvalue scale of a convolution layer of TB-Net is C×L, L is the eigenvalue length, that is, the number of eigenvalues ​​of a single channel, then after L clock cycles, the calculation of all convolution slices is completed;

[0022] Step 2-4: The PE array can process 8 channels of 8 convolution kernels at most simultaneously. When the number of convolution kernels or the number of channels of a TB-Net layer exceeds 8, we need to process it in blocks. When processing in blocks, weight blocks with the same convolution kernel but different channels are calculated first. After all weight blocks with the same convolution kernel but different channels are processed, the next batch of weight blocks with different convolution kernels are processed.

[0023] Design a custom simple instruction set as described in step 3 and write TB-Net into an instruction program. The specific process is as follows:

[0024] Customized simple instruction set, including four modes: LOAD_B / LOAD_X / CONV_W / READ_Y. It is necessary to design instruction decoder and instruction memory according to the customized instruction set. Then, the instruction program is written according to the algorithm using the instruction set so that the algorithm can be implemented on hardware, and the instruction program is stored in the instruction memory.

[0025] LOAD_B: Loads bias data stored in DRAM into the on-chip bias cache; the operand contains the amount of data to be loaded;

[0026] LOAD_X: Loads input eigenvalue data into the on-chip bias buffer; the operand contains the amount of data loaded;

[0027] CONV_W: loads the weight data stored in DRAM into the weight cache module in the PE array and performs convolution calculations; the operands include the amount of data, the number of block calculations, the length of the convolution kernel, the step size, the padding, and whether it is Relu;

[0028] READ_Y: Read the eigenvalue in the on-chip eigenvalue cache; the operand contains the data size of the eigenvalue to be read.

[0029] Step 4 describes quantizing the weights and bias data of TB-Net into 16-bit signed integer data. The specific steps are: using a symmetric pseudo-quantization scheme, pseudo-quantizing before the input of each weight and after the output of the activation function, quantizing the floating-point number to a fixed-point number, and then dequantizing it to a floating-point number, and using the floating-point value of the error generated in this process for forward operation; the pseudo-quantization operation can make the distribution of weights and activation values ​​more uniform and the variance smaller, which is smaller than the precision loss of direct post-quantization. Secondly, it can control the output of each layer within a certain range, which is more helpful for overflow processing.

[0030] The circuit function verification described in step 5 specifically includes the following steps: using the quantized data to implement the forward reasoning of the neural network using software and hardware respectively, and comparing whether the output results are consistent.

[0031] The present invention utilizes fixed weights and pulse arrays to increase the number of data multiplexing times, significantly reduce the amount of data transmission, and can adapt to one-dimensional convolutions of various scales, thereby realizing a low-power and high-speed hardware accelerator for DoA called TB-Net. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The network structure of TB-Net for DoA estimation.

[0033] Figure 2 Architecture of the TB-Net accelerator for DoA estimation.

[0034] Figure 3 This is the working principle of the PE unit.

[0035] Figure 4 This is the PE unit structure diagram. DETAILED DESCRIPTION

[0036] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0037] The present invention is a TB-Net hardware acceleration implementation method for DoA estimation. The VCS+Verdi joint simulation platform is used. The input is 120×2 16-bit data. The specific network structure is as follows Figure 1 shown.

[0038] Table 1. Network structure of TB-Net

[0039]

[0040]

[0041] The present invention mainly comprises the following steps:

[0042] Step 1: Design the architecture of the TB-Net accelerator for DoA estimation. Figure 2 shown.

[0043] Step 1-1: Design the data path from the host to the accelerator, store the weight and bias data into DRAM through the PCI-E data path, and be able to read the calculation results in the accelerator.

[0044] Step 1-2: Design the convolution calculation module PE array; suppose the PE array has 8*8 PE units, a single PE unit can process a single-channel convolution calculation of a convolution kernel, each column of PE corresponds to the convolution calculation of different channels of the same convolution kernel, and each row of PE corresponds to the convolution calculation of the same channel of different convolution kernels. Then the entire PE array can perform 8-channel convolution operations of 8 convolution kernels at the same time, and the calculation results of each column of PE are added to obtain the convolution result of eight channels; for the convolution layer calculation with the number of convolution kernels and the number of channels less than 8, the output eigenvalues ​​are added with the bias and stored in the eigenvalue cache; for the convolution layer calculation with the number of convolution kernels or the number of channels greater than 8, the convolution kernels need to be batched;

[0045] For example, for a convolution calculation with 16 convolution kernels and 16 channels, it needs to be divided into four blocks:

[0046] M 0-7 C 0-7 ,M 0-7 C 8-15 ,M 8-15 C0-7 ,M 8-15 C 8-15 ;

[0047] Among them, M 0-7 C 0-7 represents the weight blocks of channels 0 to 7 of the 0th to 7th convolution kernels, and the rest are represented in the same way as above; we use the first batch (M 0-7 C 0-7 ,M 8-15 C 0-7 ), the convolution result is added with the bias to obtain the output eigenvalue, and it is stored in the eigenvalue cache; for the non-first batch of channels (M 0-7 C 8-15 ,M 8-15 C 8-15 ), add the convolution result to the convolution result of the previous channel of the same convolution kernel in the eigenvalue cache to obtain the output eigenvalue, and store it in the eigenvalue cache.

[0048] Step 1-3: Design a single PE unit in step 1-2. Its working principle diagram is as follows: Figure 3 As shown in the figure, a single PE contains a weight storage module with an address depth of 8, 8 multipliers and 8 accumulators, and supports one-dimensional convolution calculations with a convolution kernel length of 1 to 8. Its structure diagram is shown in the figure. Figure 4 As shown in the figure, first load the corresponding weight data in DRAM into PE to fix the weight, then load a feature value data from the feature value cache into PE in each clock cycle, multiply the feature value with the weight value fixed by PE at the same time, and accumulate it with the product of the previous clock cycle, and output the corresponding output feature value of the convolution calculation at the corresponding time.

[0049] For example Figure 3 , clock cycle T0-T4, first input a single-channel 1×5 convolution kernel weight to the PE unit:

[0050] [W0,W1,W2,W3,W4],

[0051] It needs the eigenvalue of the corresponding channel:

[0052] [X0,X1,X2…X59]

[0053] Perform convolution calculation; in clock cycle T5, the input X0 to the PE unit is multiplied by [W0, W1, W2, W3, W4] respectively to obtain:

[0054] [W0X0,W1X0,W2X0,W3X0,W4X0];

[0055] In clock cycle T6, the input X1 to the PE unit is multiplied by [W0, W1, W2, W3, W4] respectively to obtain:

[0056] [W0X1,W1X1,W2X1,W3X1,W4X1];

[0057] And compared with the previous clock:

[0058] [W0X0,W1X0,W2X0,W3X0,W4X0];

[0059] Adding up the misalignments, we get:

[0060] [W0X0+W1X1,W1X0+W2X1,W2X0+W3X1,W3X0+W4X1,W4X0];

[0061] Similarly, in each clock cycle, an input feature value is multiplied by the weight value stored in PE, and the obtained product is accumulated with the product of the previous clock cycle in a staggered manner; since Padding = 2, the first convolution result is obtained in clock cycle T7:

[0062] Y0=W2X0+W3X1+W4X2;

[0063] When the convolution step length Stride = 1, the second convolution result is obtained in clock cycle T8:

[0064] Y1=W1X0+W2X1+W3X2+W4X3;

[0065] When the convolution step length Stride = 2, the second convolution result is obtained in clock cycle T9:

[0066] Y2=W0X0+W1X1+W2X2+W3X3+W4X4;

[0067] Since the number of input eigenvalues ​​is 60 and Padding=2, the reading of input eigenvalues ​​is stopped at clock cycles T65 and T66, and the product is set to 0 to obtain the final convolution results Y58 and Y59.

[0068] Step 1-4: Design an on-chip cache to temporarily store input eigenvalues, bias data, and intermediate results. For storing intermediate results, the eigenvalue cache will be divided into two parts: eigenvalue cache 1 ( Figure 2 SRAM_0Feature) and feature value cache 2 ( Figure 2SRAM_1Feature in the convolutional neural network). For one layer of calculation in the convolutional neural network, eigenvalue cache 1 is used to store the input eigenvalues ​​of the convolution calculation in this layer, and eigenvalue cache 2 is used to store the output eigenvalues ​​of the convolution calculation in this layer. For the next layer of convolution calculation, the output eigenvalues ​​of the previous layer stored in eigenvalue cache 2 are the input eigenvalues ​​of the convolution calculation in this layer, and eigenvalue cache 1 is used to store the output eigenvalues. If there is another layer of convolution calculation, the function conversion between eigenvalue cache 1 and eigenvalue cache 2 will be similar. Divide eigenvalue cache 1 and eigenvalue cache 2 into 8 equal parts to store eigenvalues ​​of different channels. The data output end of each cache corresponds to the input eigenvalue end of a row of PE arrays in step 1-2, and the data input end of each cache corresponds to the output eigenvalue end of a row of PE arrays in step 1-2.

[0069] Step 1-5: Design the control module, including the clock generator for controlling the convolution step size, the convolution padding generator, the flag signal generator during data transmission and calculation, and a series of address pointer generators.

[0070] Step 2: Design the data flow and write a script that can reorder the weights and biases of TB-Net.

[0071] Step 2-1: Read C×M×W×H weight data from DRAM and store them in the PE array. C is the number of convolution kernel channels, M is the number of convolution kernels, W is the width of the convolution kernel, and H is the height of the convolution kernel. The convolution kernel width of TB-Net used for DoA estimation is 1, so it can be simplified to C×M×H weight data.

[0072] Step 2-2: Read C×W input eigenvalues ​​from the eigenvalue cache every clock cycle. C is the number of input eigenvalue channels, which is the same as the number of convolution kernel channels, and W is the input eigenvalue width. The input eigenvalue width of TB-Net used for DoA estimation is 1, so it can be simplified to read C input eigenvalues ​​for each clock cycle. These C input eigenvalues ​​are multiplied by the C×M×H weights stored in the PE array to obtain C×M×H intermediate results. For a single PE, an input eigenvalue is multiplied by the H weights stored in it to obtain H intermediate results.

[0073] Step 2-3: In each clock cycle, the H intermediate results obtained in a single PE unit are staggered and accumulated with the intermediate results obtained in the previous cycle. After H clock cycles, the convolution result of the first convolution slice mapped on the eigenvalue can be obtained, and the convolution result of the second convolution slice can be obtained in the next clock cycle. Similarly, if the input eigenvalue scale of a convolution layer of TB-Net is C×L, L is the eigenvalue length, that is, the number of eigenvalues ​​of a single channel, then after L clock cycles, the calculation of all convolution slices is completed.

[0074] Step 2-4: The PE array can process 8 channels of 8 convolution kernels at most simultaneously. When the number of convolution kernels or the number of channels of a TB-Net layer exceeds 8, we need to process it in blocks. When processing in blocks, weight blocks with the same convolution kernel but different channels are calculated first. After all weight blocks with the same convolution kernel but different channels are processed, the next batch of weight blocks with different convolution kernels are processed.

[0075] For example, for a convolution calculation with 16 convolution kernels and 16 channels, it needs to be divided into four blocks:

[0076] M 0-7 C 0-7 ,M 0-7 C 8-15 ,M 8-15 C 0-7 ,M 8-15 C 8-15 .

[0077] The batch processing order is

[0078] M 0-7 C 0-7 →M 0-7 C 8-15 →M 8-15 C 0-7 →M 8-15 C 8-15 .

[0079] The custom simple instruction set described in step 3 includes four modes: LOAD_B / LOAD_X / CONV_W / READ_Y. The instruction decoder and instruction memory need to be designed according to the custom instruction set. Then, the instruction program is written according to the algorithm using the instruction set so that the algorithm can be implemented on the hardware, and the instruction program is stored in the instruction memory.

[0080] LOAD_B: Loads bias data stored in DRAM into the on-chip bias cache. The operand contains the amount of data to be loaded;

[0081] LOAD_X: Loads input eigenvalue data into the on-chip bias buffer. The operand contains the amount of data to be loaded;

[0082] CONV_W: Load the weight data stored in DRAM into the weight cache module in the PE array and perform convolution calculations. The operands include the amount of data, the number of block calculations, the length of the convolution kernel, the step size, the padding, and whether it is Relu.

[0083] READ_Y: Read the characteristic value in the on-chip characteristic value buffer. The operand contains the data size of the characteristic value to be read.

[0084] The quantization step described in step 4 is to use a symmetric pseudo-quantization scheme to perform pseudo-quantization before the input of each weight and after the output of the activation function, quantize the floating point number to a fixed point number, and then dequantize it to a floating point number, and use the floating point value of the error generated in this process for forward operation. The pseudo-quantization operation can make the distribution of weights and activation values ​​more uniform and the variance smaller, and the precision loss can be smaller than that of direct post-quantization. Secondly, it can control the output of each layer within a certain range, which is more helpful for overflow processing.

[0085] Pseudo-quantization scheme:

[0086] The project uses signed number calculation, so the quantization scheme is symmetric quantization. The quantization scheme is as follows:

[0087] For weights:

[0088]

[0089] round(x)=clamp(-2 bit-1 ,2 bit-1 -1,x)

[0090]

[0091]

[0092] s is the quantization factor, r is the value before quantization, and q is the value after quantization.

[0093] For activation values:

[0094] Use EMA on the activation values ​​of each layer to dynamically obtain the range:

[0095] moving max =moving max *momenta+max(abs(r))*(1-momenta)

[0096]

[0097] round(x)=clamp(-2 bit-1 ,2bit-1 -1,x)

[0098]

[0099]

[0100] Among them, s is the quantization factor, r is the value before quantization, q is the value after quantization, and momenta is the sliding index.

[0101] Folding BN (batch normalization) layers. Unlike training, where batch normalization is a separate module, in actual inference, batch normalization parameters are "folded" into the weights and biases of the convolutional or fully connected layers to improve efficiency. Folding the parameters after software training can accelerate the hardware.

[0102] The circuit function verification step described in step 5 is to use the quantized data to implement forward reasoning of the neural network on the pytorch platform and the VCS+Verdi platform respectively, and compare whether the output results of the software and hardware are consistent.

[0103] Take the comparison of software and hardware results of the first convolutional layer of TB-Net as an example.

[0104] First use the quantized data for software simulation and record the calculation results of the first layer of convolution for subsequent comparison with the hardware results.

[0105] Then write a program using a custom instruction set:

[0106] LOAD_B / / Load the weights required by TB-Net into the on-chip cache;

[0107] LOAD_X / / Load the input feature values ​​of TB-Net into the on-chip cache;

[0108] CONV_W / / Load the weights of the first layer of TB-Net into the PE array and perform the first layer of convolution calculation;

[0109] READ_Y / / Read the convolution calculation results of the first layer;

[0110] Since the design of the custom instruction set has a certain degree of flexibility, the operands after each instruction type are not listed here. The written instruction program is binary machine code.

[0111] The instruction program, multiple sets of input feature values, quantized weights and biases, and the first layer of convolution calculation results of software simulation are stored in the off-chip ROM used only for simulation through the $readmem function, and then the stimulus is given to store the instruction program in the on-chip instruction memory, and then the accelerator runs the program to start hardware simulation. The convolution results of the first layer are stored in a text file at READ_Y through the $fopen / $fwrite / $fclose function, and compared with the first layer of convolution calculation results of software simulation to calculate the error. If the error is within the indicator range, the function verification is passed.

[0112] At an operating frequency of 100 MHz, simulation results show that it only takes 1.2 ms to perform a forward inference of the TB-Net using the accelerator, and real-time DoA estimation can be achieved in most DoA estimation application scenarios.

Claims

1. A TB-Net hardware acceleration implementation method for DoA estimation, characterized in that: The specific steps are: Step 1: Design the architecture of the TB-Net accelerator for DoA estimation: Step 2: Design the data flow and write a script that can reorder the weights and biases of TB-Net; Step 3: Design a custom simple instruction set and write TB-Net into an instruction program; Step 4: Quantize the weight and bias data of TB-Net into 16-bit signed integer data; Step 5: Verify circuit function and compare software and hardware results; The specific steps of the accelerator architecture design in step 1 are as follows: Step 1-1: Design the data path from the host to the accelerator, store the weight and bias data into DRAM through the PCI-E data path, and read the calculation results from the accelerator; Step 1-2: Design the convolution calculation module PE array; Assume that the PE array has 8*8 PE units, a single PE unit is used to process a single-channel convolution calculation of a convolution kernel, each column of PE corresponds to the convolution calculation of different channels of the same convolution kernel, and each row of PE corresponds to the convolution calculation of the same channel of different convolution kernels; then the entire PE array can perform 8-channel convolution operations of 8 convolution kernels at the same time, and the calculation results of each column of PE are added to obtain the convolution result of eight channels; for the convolution layer calculation with the number of convolution kernels and the number of channels less than 8, the output eigenvalues ​​are added with the bias and stored in the eigenvalue cache; for the convolution layer calculation with the number of convolution kernels or the number of channels greater than 8, batch processing is performed; the number of convolution kernels and the number of channels in each batch shall not exceed 8; then batch processing is performed in the same way as above; Step 1-3: Design a single PE unit in step 1-2; A single PE contains a weight storage module with an address depth of 8, 8 multipliers and 8 accumulators, supporting one-dimensional convolution calculations with convolution kernel lengths of 1 to 8. First, the corresponding weight data in DRAM is loaded into the PE to fix the weight, and then a feature value data is loaded from the feature value cache to the PE in each clock cycle. The feature value is multiplied with the weight value fixed by the PE at the same time, and accumulated with the product of the previous clock cycle, and the output feature value of the corresponding convolution calculation is output at the corresponding time. Step 1-4: Design an on-chip cache to temporarily store input feature values, bias data, and intermediate results; For the storage of intermediate results, the eigenvalue cache will be equally divided into two parts: the first eigenvalue cache and the second eigenvalue cache; For one layer of convolutional network, the first eigenvalue cache is used to store the input eigenvalues ​​of the convolution calculation of this layer, and the second eigenvalue cache is used to store the output eigenvalues ​​of the convolution calculation of this layer; for the next layer of convolution calculation, the output eigenvalues ​​of the previous layer stored in the second eigenvalue cache are the input eigenvalues ​​of the convolution calculation of this layer, and the first eigenvalue cache is used to store the output eigenvalues; if there is a next layer of convolution calculation, the function conversion between the first eigenvalue cache and the second eigenvalue cache is carried out, and so on; the first eigenvalue cache and the second eigenvalue cache are divided into 8 equal parts respectively, which are used to store the eigenvalues ​​of different channels; the data output end of each cache corresponds to the input eigenvalue end of a row of PE arrays in step 1-2, and the data input end of each cache corresponds to the output eigenvalue end of a row of PE arrays in step 1-2; Step 1-5: Design the control module, including the clock generator for controlling the convolution step size, the convolution padding generator, the flag signal generator during data transmission and calculation, and a series of address pointer generators; Design data flow as described in step 2, write a script that can reorder the weights and biases of TB-Net, the steps are to store the weights and biases in DRAM according to the convolution calculation order of the PE array, and when performing convolution calculation, the accelerator reads data from DRAM according to the sequential pointer; the convolution calculation order of the PE array is: Step 2-1: Read C×M×W×H weight data from DRAM and store them in the PE array; C is the number of convolution kernel channels, M is the number of convolution kernels, W is the width of the convolution kernel, and H is the height of the convolution kernel; the convolution kernel width of TB-Net used for DoA estimation is 1, so it is simplified to C×M×H weight data; Step 2-2: Read C×W input eigenvalues ​​from the eigenvalue cache in each clock cycle; C is the number of input eigenvalue channels, which is the same as the number of convolution kernel channels, and W is the input eigenvalue width; The input eigenvalue width of TB-Net used for DoA estimation is 1, so it is simplified to read C input eigenvalues ​​for each clock cycle; these C input eigenvalues ​​are multiplied by C×M×H weights stored in the PE array to obtain C×M×H intermediate results; for a single PE, one input eigenvalue is multiplied by its H stored weights to obtain H intermediate results; Step 2-3: In each clock cycle, the H intermediate results obtained in a single PE unit are staggered and accumulated with the intermediate results obtained in the previous cycle. After H clock cycles, the convolution result of the first convolution slice mapped on the eigenvalue is obtained; the convolution result of the second convolution slice is obtained in the next clock cycle; and so on. If the input eigenvalue scale of a convolution layer of TB-Net is C×L, L is the eigenvalue length, that is, the number of eigenvalues ​​of a single channel, then after L clock cycles, the calculation of all convolution slices is completed; Step 2-4: The PE array processes 8 channels of 8 convolution kernels at most simultaneously. When the number of convolution kernels or the number of channels of a TB-Net layer exceeds 8, it is processed in blocks. When processing in blocks, weight blocks with the same convolution kernel but different channels are calculated first. After all weight blocks with the same convolution kernel but different channels are processed, the next batch of weight blocks with different convolution kernels are processed. Design a custom simple instruction set as described in step 3 and write TB-Net into an instruction program, where: A custom simple instruction set is created, including four modes: LOAD_B, LOAD_X, CONV_W, and READ_Y. The instruction decoder and instruction memory are designed based on the custom instruction set. Then, the instruction program is written based on the algorithm using the instruction set so that the algorithm can be implemented on hardware, and the instruction program is stored in the instruction memory. Among them: LOAD_B: Loads bias data stored in DRAM into the on-chip bias cache; the operand contains the amount of data to be loaded; LOAD_X: Loads input eigenvalue data into the on-chip bias buffer; the operand contains the amount of data loaded; CONV_W: loads the weight data stored in DRAM into the weight cache module in the PE array and performs convolution calculations; the operands include the amount of data, the number of block calculations, the length of the convolution kernel, the step size, the padding, and whether it is Relu; READ_Y: read the eigenvalues ​​in the on-chip eigenvalue cache; the operand contains the data volume of the read eigenvalues; Step 4 describes quantizing the weight and bias data of TB-Net into 16-bit signed integer data. The specific steps are: using a symmetric pseudo-quantization scheme, pseudo-quantizing before the input of each weight and after the output of the activation function, quantizing the floating-point number to a fixed-point number, and then dequantizing it to a floating-point number, and using the floating-point value of the error generated in this process for forward operation; The symmetric pseudo-quantization scheme is specifically as follows: For weights: round(x)=clamp(-2 bit-1 ,2 bit-1 -1,x) s is the quantization factor, r is the value before quantization, and q is the value after quantization; For activation values: Use EMA on the activation values ​​of each layer to dynamically obtain the range: moving max =moving max *momenta+max(abs(r))*(1-momenta) round(x)=clamp(-2 bit-1 ,2 bit-1 -1,x) Among them, s is the quantization factor, r is the value before quantization, q is the value after quantization, and momenta is the sliding index; Folding BN is a batch normalization layer that "folds" the batch normalization parameters into the weights and biases of the convolutional or fully connected layers to improve efficiency.

2. The TB-Net hardware acceleration implementation method for DoA estimation according to claim 1, characterized in that: The circuit function verification described in step 5 includes using the quantized data to implement forward reasoning of the neural network using software and hardware respectively, and comparing whether the output results are consistent.

Images