FPGA-based mixed-precision data frequency domain convolution acceleration method and system

Abstract

The application provides an FPGA-based mixed-precision data frequency domain convolution acceleration method and system, which comprises the following steps: obtaining 8-bit data and 16-bit data; routing each 16-bit data to a first storage location for caching after being equally divided; packing two 8-bit data caching results or one 16-bit data caching result to a second storage location for storage; then performing frequency domain convolution calculation, accumulation operation and data recovery processing on the 8-bit data or 16-bit data to obtain 8-bit data calculation results or 16-bit data calculation results, and completing the frequency domain convolution calculation of mixed-precision data; the application can be compatible with the frequency domain convolution calculation of 8-bit and 16-bit data, and can effectively improve the utilization efficiency of a DSP and the operation speed of frequency domain convolution.

Description

Technical Field

[0001] This invention relates to the fields of digital signal processing and frequency domain convolution calculation technology, and more specifically, to an FPGA-based method and system for accelerating mixed-precision data frequency domain convolution. Background Technology

[0002] Image recognition is a relatively mature research task in computer vision, and convolutional neural networks (CNNs) perform exceptionally well in handling such tasks. However, when deploying more complex CNNs on hardware accelerators, it becomes difficult to implement them due to the limited resources of the hardware platform, such as FPGAs and ASICs. To solve this problem, the only solution is to optimize the convolution computation algorithm. Among these optimizations, the Fast Fourier Transform (FFT) algorithm was proposed, which essentially transforms traditional spatial domain convolution computation into frequency domain convolution computation.

[0003] For frequency domain convolution calculations, because it uses the simple EWMM (Element-Wise Multiplication) operation to replace the traditional spatial convolution operation, the computational complexity can be effectively reduced. When the number of calculations is effectively reduced, the difficulty of hardware deployment will also be reduced.

[0004] Meanwhile, the reduction in resources also facilitates the deployment of neural networks on hardware. Low-precision convolutional neural networks (such as 8-bit CNNs), as a method that can effectively reduce memory consumption and bandwidth, have gained widespread support from many deep learning frameworks, such as TensorFlow, PyTorch, NVIDIA TensorRT, and Xilinx DNNDK. Therefore, low-precision convolutional neural networks are also becoming popular in the design of hardware accelerators.

[0005] Currently, some FPGA-based frequency domain convolutional neural network accelerators have adopted different acceleration schemes to reduce computational complexity, including: using FFT overlap and added convolution (FFT-OVA-Conv), using overlap addition, and using double buffering and shared memory, etc. However, these FPGA accelerators all use high data precision types (such as float), so their computational data precision is very limited.

[0006] Current technology discloses a lightweight convolutional neural network reconfigurable deployment method based on FPGA. During deployment, it adapts to FPGA platforms with different resource conditions by configuring parameters such as the number of convolutional window columns and the number of output channels. Before running, it adapts to different network layers by configuring different convolutional kernel sizes, pooling strides, input and output feature map sizes, and the number of input and output channels. It constructs a lightweight convolutional neural network accelerator based on FPGA to achieve reconfigurable network deployment. The lightweight convolutional neural network accelerator includes an off-chip DDR data interaction unit, an on-chip memory buffer processing unit, a computing engine, and a central control unit. The existing technology uses a limited number of data precision types and bit widths, which greatly affects the accuracy of the results. It cannot reduce the size and computational complexity of the model while maintaining model accuracy. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, which uses data with limited and singular bit widths of precision types and cannot balance the accuracy and complexity of the model, this invention provides a method and system for accelerating mixed-precision data frequency domain convolution based on FPGA. This method can process data of multiple precision types simultaneously, effectively improving the utilization efficiency of DSP and the operation speed of frequency domain convolution.

[0008] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0009] An FPGA-based method for accelerating mixed-precision data frequency domain convolution includes the following steps:

[0010] S1: Obtain several 8-bit and 16-bit data;

[0011] S2: Directly route each 8-bit data to the first storage location for caching to obtain an 8-bit data caching result; divide each 16-bit data equally and route it to the first storage location for caching to obtain a 16-bit data caching result;

[0012] S3: Select two 8-bit data cache results from the first storage location, pack them into the second storage location for storage, and obtain an 8-bit data storage result; or select a 16-bit data cache result from the first storage location, directly input it into the second storage location for storage, and obtain a 16-bit data storage result;

[0013] S4: Perform frequency domain convolution calculation on the 8-bit data storage result, and perform offset correction on the frequency domain convolution calculation result of the 8-bit data storage result to obtain the 8-bit data frequency domain convolution result; or perform frequency domain convolution calculation on the 16-bit data storage result to obtain the 16-bit data frequency domain convolution result.

[0014] S5: Perform an accumulation operation on the frequency domain convolution result of 8-bit data or the frequency domain convolution result of 16-bit data to obtain the accumulated result of 8-bit data or the accumulated result of 16-bit data;

[0015] S6: The obtained 8-bit or 16-bit data accumulation result is processed by a preset algorithm to recover the data, and the 8-bit or 16-bit data calculation result is obtained, thus completing the frequency domain convolution calculation of mixed precision data.

[0016] Preferably, the specific method of step S2 is as follows:

[0017] The bit width of the first storage location is 8 bits;

[0018] For each 8-bit data, it is directly routed to the ordered position in the first storage location for caching, thus obtaining the 8-bit data caching result;

[0019] For each 16-bit data, divide each 16-bit data into the first 8 bits and the last 8 bits. Then, route the first 8 bits and the last 8 bits to the ordered positions in the first storage location for caching to obtain the 16-bit data caching result.

[0020] Preferably, the specific method of step S3 is as follows:

[0021] The second storage location has a bit width of 27 bits, including bits 0 to 26;

[0022] Two 8-bit data cache results are selected from the first storage location and packaged into the second storage location. One 8-bit data cache result is stored in bits 0 to 7 of the second storage location, and the other 8-bit data cache result is stored in bits 18 to 25 of the second storage location. Bit 26 is used as the sign bit. Bits 8 to 17 are filled with zeros to obtain the 8-bit data storage result.

[0023] Alternatively, select a 16-bit data cache result and directly input it into the second storage location. The first 8 bits and the last 8 bits are stored in bits 0 to 15 of the second storage location, and the 16th bit is used as the sign bit. Bits 17 to 26 are filled with zeros to obtain the 16-bit data storage result.

[0024] Preferably, the specific method of step S4 is as follows:

[0025] The frequency domain convolution calculation is performed on the 8-bit data storage result. The two 8-bit data storage results are input into data registers A and C respectively to perform convolution multiplication calculations. The convolution multiplication calculation results of the 8-bit data storage result are offset and corrected to obtain the 8-bit data frequency domain convolution result.

[0026] Alternatively, frequency domain convolution calculation can be performed on the 16-bit data storage result. The two 16-bit data storage results are input into data registers A and C respectively to perform convolution multiplication calculations to obtain the 16-bit data frequency domain convolution result.

[0027] Store the obtained 8-bit data frequency domain convolution result or 16-bit data frequency domain convolution result into data register E;

[0028] The data registers A and C have a bit width of 27 bits, and the data register E has a bit width of 48 bits.

[0029] Preferably, the convolution multiplication calculation specifically involves:

[0030] Perform addition or subtraction operations on the two 8-bit or 16-bit data storage results in data registers A and C to obtain the first intermediate result, denoted as: A±C;

[0031] Perform a convolution multiplication calculation on the first intermediate result and the convolution kernel in the preset convolution kernel register B, denoted as: (A±C)×B;

[0032] The preset convolution kernel register B has a width of 18 bits and is used to store convolution kernels of 8-bit or 16-bit precision.

[0033] Preferably, the specific method for performing addition or subtraction operations on the two 8-bit or 16-bit data storage results in data registers A and C is as follows:

[0034] The calculation method is controlled using a preset inmode control signal;

[0035] When the inmode control signal is "+", the two 8-bit data storage results or 16-bit data storage results in data registers A and C are added together to obtain the first intermediate result, denoted as: A+C;

[0036] When the inmode control signal is "-", the two 8-bit data storage results or 16-bit data storage results in data registers A and C are subtracted to obtain the first intermediate result, denoted as AC.

[0037] Preferably, the specific method for offset correction of the convolution multiplication calculation result of the 8-bit data storage result is as follows:

[0038] The convolution multiplication result of the 8-bit data storage result is input into the preset shift register D, and the value of shift register D is determined. The result of the convolution multiplication calculation is offset and corrected according to the value of shift register D to obtain the frequency domain convolution result of 8-bit data, denoted as: (A±C)×B+D;

[0039] The shift register D has a bit width of 48 bits;

[0040] The specific method for determining the value of shift register D is as follows:

[0041] When the inmode control signal is "+": If the data in bits 0 to 7 of data registers A and C are all negative, then the value of shift register D is the value obtained by shifting the data in kernel register B left by 9 bits and then inverting it, denoted as: D = -(B << 9);

[0042] If the data in bits 0-7 of data registers A and C is negative and non-negative respectively, then the value of shift register D is the value obtained by shifting the data in kernel register B left by 8 bits and then inverting it, denoted as: D = -(B << 8); otherwise, the value of shift register D is 0.

[0043] When the inmode control signal is "-": if the data in bits 0 to 7 of data register A is negative and the data in bits 0 to 7 of data register C is non-negative, then the value of shift register D is the value obtained by shifting the data in kernel register B left by 8 bits and then inverting it, denoted as: D = -(B << 8);

[0044] If the data in bits 0-7 of data register A is non-negative and the data in bits 0-7 of data register C is negative, then the value of shift register D is the value obtained by shifting the data in kernel register B left by 8 bits, denoted as: D = (B << 8); otherwise, the value of shift register D is 0.

[0045] Preferably, in step S5, the 8-bit data frequency domain convolution result or the 16-bit data frequency domain convolution result is accumulated to obtain an 8-bit data accumulation result or a 16-bit data accumulation result. The specific method is as follows:

[0046] The 8-bit data frequency domain convolution result or the 16-bit data frequency domain convolution result has a bit width of 16 bits.

[0047] The obtained 8-bit data frequency domain convolution result is divided equally. The first 8 bits of the 8-bit data frequency domain convolution result are used as the low-order data, and the last 8 bits of the 8-bit data frequency domain convolution result are used as the high-order data. The low-order data and the high-order data are accumulated separately to obtain the 8-bit data accumulation result.

[0048] The obtained 16-bit data frequency domain convolution results are directly accumulated to obtain the 16-bit data accumulation result.

[0049] Preferably, in step S6, the obtained 8-bit data accumulation result or 16-bit data accumulation result is processed by a preset algorithm to recover the data, thereby obtaining the 8-bit data calculation result or the 16-bit data calculation result. The specific method is as follows:

[0050] The obtained 8-bit or 16-bit data accumulation results are processed by a preset algorithm to recover the data. The real and imaginary parts of the 8-bit or 16-bit data accumulation results are calculated respectively to obtain the complex number form of the 8-bit or 16-bit data calculation results.

[0051] This invention also provides an FPGA-based mixed-precision data frequency domain convolution acceleration system, which applies the above-mentioned FPGA-based mixed-precision data frequency domain convolution acceleration method, including:

[0052] Data acquisition module: used to acquire several 8-bit and 16-bit data;

[0053] Data routing module: used to directly route each 8-bit data to the first storage location for caching, obtaining an 8-bit data caching result; and to divide each 16-bit data equally and route it to the first storage location for caching, obtaining a 16-bit data caching result.

[0054] Data preprocessing module: used to select two 8-bit data cache results from the first storage location, package them into the second storage location for storage, and obtain an 8-bit data storage result; or select a 16-bit data cache result from the first storage location and directly input it into the second storage location for storage, and obtain a 16-bit data storage result;

[0055] Frequency domain convolution module: used to perform frequency domain convolution calculation on 8-bit data storage results, and to perform offset correction on the frequency domain convolution calculation results of 8-bit data storage results to obtain 8-bit data frequency domain convolution results; or to perform frequency domain convolution calculation on 16-bit data storage results to obtain 16-bit data frequency domain convolution results;

[0056] Data accumulation module: used to accumulate the frequency domain convolution results of 8-bit data or 16-bit data to obtain the accumulated 8-bit data result or the accumulated 16-bit data result;

[0057] Data recovery module: Used to perform data recovery processing on the obtained 8-bit data accumulation result or 16-bit data accumulation result through a preset algorithm to obtain the 8-bit data calculation result or 16-bit data calculation result, and complete the frequency domain convolution calculation of mixed precision data.

[0058] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0059] This invention provides a method and system for accelerating mixed-precision data frequency domain convolution based on FPGA. The method acquires several 8-bit and 16-bit data points; each 8-bit data point is directly routed to a first storage location for caching, obtaining an 8-bit data cache result; each 16-bit data point is equally divided and routed to the first storage location for caching, obtaining a 16-bit data cache result; two 8-bit data cache results are selected from the first storage location and packaged into a second storage location for storage, obtaining an 8-bit data storage result; or one 16-bit data cache result is selected from the first storage location and directly input into the second storage location for storage, obtaining a 16-bit data storage result. The 8-bit data storage result is subjected to frequency domain convolution calculation, and the frequency domain convolution calculation result of the 8-bit data storage result is offset and corrected to obtain the 8-bit data frequency domain convolution result; or the 16-bit data storage result is subjected to frequency domain convolution calculation to obtain the 16-bit data frequency domain convolution result; the 8-bit data frequency domain convolution result or the 16-bit data frequency domain convolution result is accumulated to obtain the 8-bit data accumulation result or the 16-bit data accumulation result; the obtained 8-bit data accumulation result or the 16-bit data accumulation result is processed by a preset algorithm to obtain the 8-bit data calculation result or the 16-bit data calculation result, thus completing the frequency domain convolution calculation of mixed precision data;

[0060] This invention utilizes the characteristics of frequency domain convolution to be compatible with frequency domain convolution calculations for both 8-bit and 16-bit precision data types. It can simultaneously process multiple mixed precision data types, effectively improving the utilization efficiency of DSPs in FPGAs and the operation speed of frequency domain convolution. In addition, the introduction of offset correction operations in the calculation of 8-bit data effectively improves the calculation accuracy and efficiency of 8-bit data. Attached Figure Description

[0061] Figure 1The flowchart of a mixed-precision data frequency domain convolution acceleration method based on FPGA provided in Example 1 is shown.

[0062] Figure 2 This is a schematic diagram of a mixed-precision data frequency domain convolution acceleration method based on FPGA provided in Example 2.

[0063] Figure 3 This is a schematic diagram of the data accumulation operation provided in Example 2.

[0064] Figure 4 This is a schematic diagram of the data recovery operation provided in Example 2.

[0065] Figure 5 This is a schematic diagram of frequency domain convolution calculation provided in Example 2.

[0066] Figure 6 This is a schematic diagram of the data routing operation provided in Example 2.

[0067] Figure 7 This is a diagram of the algorithm for determining the value of shift register D provided in Example 2.

[0068] Figure 8 This is a structural diagram of a mixed-precision data frequency domain convolution acceleration system based on FPGA provided in Example 3. Detailed Implementation

[0069] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0070] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0071] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0072] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0073] Example 1

[0074] like Figure 1 As shown, this embodiment provides a method for accelerating mixed-precision data frequency domain convolution based on FPGA, including the following steps:

[0075] S1: Obtain several 8-bit and 16-bit data;

[0076] S2: Directly route each 8-bit data to the first storage location for caching to obtain an 8-bit data caching result; divide each 16-bit data equally and route it to the first storage location for caching to obtain a 16-bit data caching result;

[0077] S3: Select two 8-bit data cache results from the first storage location, pack them into the second storage location for storage, and obtain an 8-bit data storage result; or select a 16-bit data cache result from the first storage location, directly input it into the second storage location for storage, and obtain a 16-bit data storage result;

[0078] S4: Perform frequency domain convolution calculation on the 8-bit data storage result, and perform offset correction on the frequency domain convolution calculation result of the 8-bit data storage result to obtain the 8-bit data frequency domain convolution result; or perform frequency domain convolution calculation on the 16-bit data storage result to obtain the 16-bit data frequency domain convolution result.

[0079] S5: Perform an accumulation operation on the frequency domain convolution result of 8-bit data or the frequency domain convolution result of 16-bit data to obtain the accumulated result of 8-bit data or the accumulated result of 16-bit data;

[0080] S6: The obtained 8-bit or 16-bit data accumulation result is processed by a preset algorithm to recover the data, and the 8-bit or 16-bit data calculation result is obtained, thus completing the frequency domain convolution calculation of mixed precision data.

[0081] In the specific implementation process, firstly, several 8-bit and 16-bit data are acquired; each 8-bit data is directly routed to the first storage location for caching, obtaining an 8-bit data caching result; each 16-bit data is equally divided and routed to the first storage location for caching, obtaining a 16-bit data caching result; from the first storage location, two 8-bit data caching results are selected, packaged, and stored in the second storage location, obtaining an 8-bit data storage result; or from the first storage location, one 16-bit data caching result is selected and directly input into the second storage location for storage, obtaining a 16-bit data storage result; the 8-bit data storage result is then processed... Frequency domain convolution calculation is performed, and the frequency domain convolution calculation result of the 8-bit data storage result is offset-corrected to obtain the 8-bit data frequency domain convolution result; or frequency domain convolution calculation is performed on the 16-bit data storage result to obtain the 16-bit data frequency domain convolution result; the 8-bit data frequency domain convolution result or the 16-bit data frequency domain convolution result is accumulated to obtain the 8-bit data accumulation result or the 16-bit data accumulation result; the obtained 8-bit data accumulation result or the 16-bit data accumulation result is processed by a preset algorithm to obtain the 8-bit data calculation result or the 16-bit data calculation result, thus completing the frequency domain convolution calculation of mixed precision data;

[0082] This method leverages the characteristics of frequency domain convolution to be compatible with frequency domain convolution calculations for both 8-bit and 16-bit precision data types. It can simultaneously process multiple mixed precision data types, effectively improving the utilization efficiency of the DSP in the FPGA and the operation speed of frequency domain convolution. In addition, the introduction of offset correction operations in the calculation of 8-bit data effectively improves the calculation accuracy and efficiency of 8-bit data.

[0083] Example 2

[0084] like Figure 2 As shown, this embodiment provides a method for accelerating mixed-precision data frequency domain convolution based on FPGA, including the following steps:

[0085] S1: Obtain several 8-bit and 16-bit data;

[0086] S2: Directly route each 8-bit data to the first storage location for caching to obtain an 8-bit data caching result; divide each 16-bit data equally and route it to the first storage location for caching to obtain a 16-bit data caching result;

[0087] The bit width of the first storage location is 8 bits;

[0088] For each 8-bit data, it is directly routed to the ordered position in the first storage location for caching, thus obtaining the 8-bit data caching result;

[0089] For each 16-bit data, divide each 16-bit data into the first 8 bits and the last 8 bits. Then, route the first 8 bits and the last 8 bits to the ordered positions in the first storage location for caching to obtain the 16-bit data caching result.

[0090] S3: Select two 8-bit data cache results from the first storage location, pack them into the second storage location for storage, and obtain an 8-bit data storage result; or select one 16-bit data cache result from the first storage location, directly input it into the second storage location for storage, and obtain a 16-bit data storage result, specifically:

[0091] The second storage location has a bit width of 27 bits, including bits 0 to 26;

[0092] Two 8-bit data cache results are selected from the first storage location and packaged into the second storage location. One 8-bit data cache result is stored in bits 0 to 7 of the second storage location, and the other 8-bit data cache result is stored in bits 18 to 25 of the second storage location. Bit 26 is used as the sign bit. Bits 8 to 17 are filled with zeros to obtain the 8-bit data storage result.

[0093] Alternatively, select a 16-bit data cache result and directly input it into the second storage location. The first 8 bits and the last 8 bits are stored in bits 0 to 15 of the second storage location, and the 16th bit is used as the sign bit. Bits 17 to 26 are filled with zeros to obtain the 16-bit data storage result.

[0094] S4: Perform frequency domain convolution calculation on the 8-bit data storage result, and perform offset correction on the frequency domain convolution calculation result of the 8-bit data storage result to obtain the 8-bit data frequency domain convolution result; or perform frequency domain convolution calculation on the 16-bit data storage result to obtain the 16-bit data frequency domain convolution result, specifically:

[0095] The frequency domain convolution calculation is performed on the 8-bit data storage result. The two 8-bit data storage results are input into data registers A and C respectively to perform convolution multiplication calculations. The convolution multiplication calculation results of the 8-bit data storage result are offset and corrected to obtain the 8-bit data frequency domain convolution result.

[0096] Alternatively, frequency domain convolution calculation can be performed on the 16-bit data storage result. The two 16-bit data storage results are input into data registers A and C respectively to perform convolution multiplication calculations to obtain the 16-bit data frequency domain convolution result.

[0097] Store the obtained 8-bit data frequency domain convolution result or 16-bit data frequency domain convolution result into data register E;

[0098] The data registers A and C have a bit width of 27 bits, and the data register E has a bit width of 48 bits.

[0099] S5: Perform an accumulation operation on the frequency domain convolution results of 8-bit data or 16-bit data to obtain an 8-bit data accumulation result or a 16-bit data accumulation result, specifically:

[0100] The 8-bit or 16-bit data frequency domain convolution results are accumulated to obtain an 8-bit or 16-bit accumulated data result. The specific method is as follows:

[0101] The 8-bit data frequency domain convolution result or the 16-bit data frequency domain convolution result has a bit width of 16 bits.

[0102] The obtained 8-bit data frequency domain convolution result is divided equally. The first 8 bits of the 8-bit data frequency domain convolution result are used as the low-order data, and the last 8 bits of the 8-bit data frequency domain convolution result are used as the high-order data. The low-order data and the high-order data are accumulated separately to obtain the 8-bit data accumulation result.

[0103] The obtained 16-bit data frequency domain convolution results are directly accumulated to obtain the 16-bit data accumulation result;

[0104] S6: Perform data recovery processing on the obtained 8-bit data accumulation result or 16-bit data accumulation result through a preset algorithm, calculate the real part and imaginary part of the 8-bit data accumulation result or 16-bit data accumulation result respectively, obtain the 8-bit data calculation result or 16-bit data calculation result in complex form, and complete the frequency domain convolution calculation of mixed precision data;

[0105] The convolution multiplication calculation is specifically as follows:

[0106] Perform addition or subtraction operations on the two 8-bit or 16-bit data storage results in data registers A and C to obtain the first intermediate result, denoted as: A±C;

[0107] Perform a convolution multiplication calculation on the first intermediate result and the convolution kernel in the preset convolution kernel register B, denoted as: (A±C)×B;

[0108] The preset convolution kernel register B has a width of 18 bits and is used to store convolution kernels of 8-bit or 16-bit precision type.

[0109] The specific method for performing addition or subtraction operations on the two 8-bit or 16-bit data storage results in data registers A and C is as follows:

[0110] The calculation method is controlled using a preset inmode control signal;

[0111] When the inmode control signal is "+", the two 8-bit data storage results or 16-bit data storage results in data registers A and C are added together to obtain the first intermediate result, denoted as: A+C;

[0112] When the inmode control signal is "-", the two 8-bit data storage results or 16-bit data storage results in data registers A and C are subtracted to obtain the first intermediate result, denoted as AC.

[0113] The specific method for offset correction of the convolution multiplication calculation result of 8-bit data storage is as follows:

[0114] The convolution multiplication result of the 8-bit data storage result is input into the preset shift register D, and the value of shift register D is determined. The result of the convolution multiplication calculation is offset and corrected according to the value of shift register D to obtain the frequency domain convolution result of 8-bit data, denoted as: (A±C)×B+D;

[0115] The shift register D has a bit width of 48 bits;

[0116] The specific method for determining the value of shift register D is as follows:

[0117] When the inmode control signal is "+": If the data in bits 0 to 7 of data registers A and C are all negative, then the value of shift register D is the value obtained by shifting the data in kernel register B left by 9 bits and then inverting it, denoted as: D = -(B << 9);

[0118] If the data in bits 0-7 of data registers A and C is negative and non-negative respectively, then the value of shift register D is the value obtained by shifting the data in kernel register B left by 8 bits and then inverting it, denoted as: D = -(B << 8); otherwise, the value of shift register D is 0.

[0119] When the inmode control signal is "-": if the data in bits 0 to 7 of data register A is negative and the data in bits 0 to 7 of data register C is non-negative, then the value of shift register D is the value obtained by shifting the data in kernel register B left by 8 bits and then inverting it, denoted as: D = -(B << 8);

[0120] If the data in bits 0-7 of data register A is non-negative and the data in bits 0-7 of data register C is negative, then the value of shift register D is the value obtained by shifting the data in kernel register B left by 8 bits, denoted as: D = (B << 8); otherwise, the value of shift register D is 0.

[0121] In the specific implementation process, several 8-bit data and 16-bit data are first acquired;

[0122] Each 8-bit data is directly routed to the first storage location for caching, resulting in an 8-bit data cache result; each 16-bit data is then divided equally and routed to the first storage location for caching, resulting in a 16-bit data cache result.

[0123] Two 8-bit data cache results are selected from the first storage location and packaged into the second storage location for storage to obtain an 8-bit data storage result; or a 16-bit data cache result is selected from the first storage location and directly input into the second storage location for storage to obtain a 16-bit data storage result.

[0124] In this embodiment, the purpose of reserving the highest bit of the input data register is to prevent overflow of the pre-addition operation in subsequent complex number calculations, so as to realize the simultaneous calculation of two 8-bit data multiplications or one 16-bit data multiplication in frequency domain convolution in a DSP.

[0125] Perform frequency domain convolution calculation on the 8-bit data storage result, and perform offset correction on the frequency domain convolution calculation result of the 8-bit data storage result to obtain the 8-bit data frequency domain convolution result; or perform frequency domain convolution calculation on the 16-bit data storage result to obtain the 16-bit data frequency domain convolution result.

[0126] like Figure 3 As shown, the frequency domain convolution results of 8-bit data or 16-bit data are accumulated to obtain the accumulated 8-bit data result or the accumulated 16-bit data result.

[0127] like Figure 4As shown, the accumulated 8-bit or 16-bit data results are processed by a preset algorithm to recover the data, and the calculated 8-bit or 16-bit data results are obtained. The frequency domain convolution calculation of mixed precision data is completed. For example, the real part of data p can be obtained by x+y, and the imaginary part of data p can be obtained by x+z, thus obtaining the complete complex output p.

[0128] like Figure 5 As shown, Figure 5 This is a schematic diagram of frequency domain convolution, which includes an input feature matrix I and convolution kernels K1 and K2. Generally, it is a three-channel input. This embodiment uses a single channel for frequency domain convolution calculation.

[0129] The input feature matrix and convolution kernel need to be transformed from real values ​​in the spatial domain to complex values ​​in the frequency domain using the FFT (Fast Fourier Transform) algorithm. Then, a frequency domain convolution multiplication calculation will be performed between the two, specifically:

[0130] Assume A=a r +a i *j is a complex activation value at a certain point on I, W1 = w r1 +w i1 *j、W2=w r2 +w i2 *j represents the complex weights corresponding to A on K1 and K2, respectively, where j represents the imaginary part of the complex number; when performing frequency domain convolution calculation, complex multiplication of A×W1 and A×W2 occurs, which can be expressed as: A×W1=(a r +a i *j)×(w r1 +w i1 *j); After simplification, we get: the real part of A×W1 is a r ×(w r1 -w i1 )+w i1 ×(a r -a i The imaginary part is a. i ×(w r1 +w i1 )+w i1 ×(a r -a i Therefore, to obtain the complex multiplication A×W1, three real multiplications are required: a r ×(w r1 -w i1 ), a i ×(w r1 +w i1 ) and w i1 ×(a r -a iSimilarly, A×W2 will also involve three real number multiplications.

[0131] Figure 2 The frequency domain convolution acceleration method is compatible with both 8-bit and 16-bit computation modes;

[0132] In 8-bit computation mode, the DSP can compute two multiplication expressions simultaneously, with a r ×(w r1 -w i1 ) and a r ×(w r2 -w i2 Let's take an example to illustrate:

[0133] Both of the above multiplication expressions have a common multiplier 'a'. r Their other multipliers are (w r1 -w i1 ) and (w r2 -w i2 Both require subtraction; since they both require 8-bit multiplication, the method in this embodiment can be used to first package a pair of 8-bit data, so that two 8-bit multiplication operations can be performed simultaneously in a DSP; first, obtain w r1 and w r2 After two 8-bit data, they are routed to the ordered location in the local buffer LB. Then, the same set of 8-bit data that needs to be processed is read and packed into the data register A.

[0134] like Figure 6 As shown, Figure 6 The data 1 and 3 in the text represent w respectively. r1 and w r2 Data 2 and 4 represent w respectively. i1 and w i2 Data 1 and 3 are stored in data register A through a packetization process. The final packetization result can be represented as {w r1 [7], w r1 [7:0],10'b0,w r2 [7:0]}, it can be seen that w at this time r1 [7:0] is the high-order entry in data register A, w r2 [7:0] is located in the low-order entry of data register A; similarly, data 2 and 4 will be in {w i1 [7], w i1 [7:0],10'b0,w i2 The format [7:0]} is packed in data register C;

[0135] Then, frequency domain convolution calculation is performed, and the common multiplier ar The result is pre-stored in the convolution kernel register B. The calculation of AC is performed via the inmode control signal, and then the calculation of (AC) × B begins. In effect, a is executed simultaneously. r ×(w r1 -w i1 ) and a r ×(w r2 -w i2 Two multiplication expressions;

[0136] It is worth noting that the result of (AC)×B is not the final output of the frequency domain convolution calculation, because the low-order terms of the data after being packed in data registers A and C are missing the sign bit. Therefore, the purpose of setting up shift register D is to eliminate the low-order term multiplication result (i.e., a). r ×(w r2 -w i2 The deviation exists;

[0137] like Figure 7 As shown, in Figure 7 The diagram shows registers A, B, and C, where A... m C m Indicates the higher-order terms (in this example, w respectively) r1 w i1 A1 and C1 represent the lower-order terms (in this example, they refer to w respectively). r2 w i2 );in addition, Figure 5 The inmode input is the calculation mode control signal for frequency domain convolution;

[0138] When the inmode control signal is "+": If the data in bits 0 to 7 of data registers A and C are all negative, then the value of shift register D is the value obtained by shifting the data in kernel register B left by 9 bits and then inverting it, denoted as: D = -(B << 9);

[0139] If the data in bits 0-7 of data registers A and C is negative and non-negative respectively, then the value of shift register D is the value obtained by shifting the data in kernel register B left by 8 bits and then inverting it, denoted as: D = -(B << 8); otherwise, the value of shift register D is 0.

[0140] When the inmode control signal is "-": if the data in bits 0 to 7 of data register A is negative and the data in bits 0 to 7 of data register C is non-negative, then the value of shift register D is the value obtained by shifting the data in kernel register B left by 8 bits and then inverting it, denoted as: D = - (B << 8).

[0141] If the data in bits 0-7 of data register A is non-negative and the data in bits 0-7 of data register C is negative, then the value of shift register D is the value obtained by shifting the data in kernel register B left by 8 bits, denoted as: D = (B << 8); otherwise, the value of shift register D is 0.

[0142] After the frequency domain convolution calculation is completed, the results need to be accumulated. To prevent overflow during the accumulation process, the two multiplication results (i.e., a) in the data register E are stored separately. r ×(w r1 -w i1 ) and a r ×(w r2 -w i2 The data is stored separately in registers with higher bit widths, and then the data for different input channels is processed separately. r ×(w r1 -w i1 ) and a r ×(w r2 -w i2 The results are summed up.

[0143] Similarly, the remaining multiplication expressions in A×W1 and A×W2 can be calculated in the same way;

[0144] After the six multiplications in A×W1 and A×W2 are completed, a data recovery operation is required to obtain the final complex number result. Taking the recovery of A×W1 as an example, only a needs to be calculated. r ×(w r1 -w i1 )+w i1 ×(a r -a i ) and a i ×(w r1 +w i1 )+w i1 ×(a r -a i This will give you the complex number result of A×W1;

[0145] In 16-bit computation mode, because the data storage bit width is limited, the DSP can only compute one multiplication expression, with a r ×(w r -w i Let's take 16-bit data as an example to illustrate; first, we will use 16-bit data w r The data is divided into two 8-bit data segments in an equal manner, then routed to the ordered position of the local cache LB, and then the 16-bit data is restored and packed into the data register A using the same memory access mode.

[0146] like Figure 6As shown, data 1 and 3 correspond to w. r Using the high 8 bits and low 8 bits, data 1 and 3 are restored to their original data and packed into data register A. The final packed result is {11'b0, w}. r [15:0]}; Similarly, data 2 and 4 correspond to data w i Finally, the result packed into data register C is {11'b0, w} i [15:0]};

[0147] The subsequent frequency domain calculations will take the common multiplier a r The data is stored in the convolution kernel register B in advance. The calculation of AC is performed through the inmode control signal, and then the calculation of (AC)×B begins.

[0148] Since this is a 16-bit calculation mode, the input data is not affected by the result deviation of the lower-order terms, so no input adjustment for D is needed. Therefore, the value of shift register D is filled with 0s, and finally the result of (AC)×B+D is input into data register E; then, the accumulation operation is performed, at which point only a needs to be added. r ×(w r -w i )+w i ×(a r -a i ) and a i ×(w r +w i )+w i ×(a r -a i After processing the three multiplication expressions, performing data restoration operations will yield the complex number result of A×W.

[0149] This method leverages the characteristics of frequency domain convolution to be compatible with frequency domain convolution calculations for both 8-bit and 16-bit precision data types. It can simultaneously process multiple mixed precision data types, effectively improving the utilization efficiency of the DSP in the FPGA and the operation speed of frequency domain convolution. In addition, the introduction of offset correction operations in the calculation of 8-bit data effectively improves the calculation accuracy and efficiency of 8-bit data.

[0150] Example 3

[0151] like Figure 8 As shown, this embodiment provides an FPGA-based mixed-precision data frequency domain convolution acceleration system, applying an FPGA-based mixed-precision data frequency domain convolution acceleration method from embodiment 1 or 2, including:

[0152] Data acquisition module 301: used to acquire several 8-bit and 16-bit data;

[0153] Data routing module 302: used to directly route each 8-bit data to the first storage location for caching, and obtain an 8-bit data caching result; and to divide each 16-bit data equally and route it to the first storage location for caching, and obtain a 16-bit data caching result.

[0154] Data preprocessing module 303: used to select two 8-bit data cache results from the first storage location, pack them into the second storage location for storage, and obtain an 8-bit data storage result; or select a 16-bit data cache result from the first storage location and directly input it into the second storage location for storage, and obtain a 16-bit data storage result;

[0155] Frequency domain convolution module 304: used to perform frequency domain convolution calculation on 8-bit data storage results, and to perform offset correction on the frequency domain convolution calculation results of 8-bit data storage results to obtain 8-bit data frequency domain convolution results; or to perform frequency domain convolution calculation on 16-bit data storage results to obtain 16-bit data frequency domain convolution results;

[0156] Data accumulation module 305: used to accumulate the frequency domain convolution result of 8-bit data or the frequency domain convolution result of 16-bit data to obtain the 8-bit data accumulation result or the 16-bit data accumulation result;

[0157] Data recovery module 306: Used to perform data recovery processing on the obtained 8-bit data accumulation result or 16-bit data accumulation result through a preset algorithm to obtain the 8-bit data calculation result or 16-bit data calculation result and complete the frequency domain convolution calculation of mixed precision data.

[0158] In the specific implementation process, firstly, the data acquisition module 301 acquires several 8-bit and 16-bit data; then, the data routing module 302 directly routes each 8-bit data to the first storage location for caching, obtaining an 8-bit data caching result; after evenly dividing each 16-bit data, it routes it to the first storage location for caching, obtaining a 16-bit data caching result; the data preprocessing module 303 selects two 8-bit data caching results from the first storage location, packages them into a second storage location for storage, obtaining an 8-bit data storage result; or selects one 16-bit data caching result from the first storage location and directly inputs it into the second storage location for storage, obtaining a 16-bit data storage result; the frequency domain convolution module 304 processes the 8-bit data... The data storage result is subjected to frequency domain convolution calculation, and the frequency domain convolution calculation result of the 8-bit data storage result is offset and corrected to obtain the 8-bit data frequency domain convolution result; or the 16-bit data storage result is subjected to frequency domain convolution calculation to obtain the 16-bit data frequency domain convolution result; the data accumulation module 305 accumulates the 8-bit data frequency domain convolution result or the 16-bit data frequency domain convolution result to obtain the 8-bit data accumulation result or the 16-bit data accumulation result; finally, the data recovery module 306 performs data recovery processing on the obtained 8-bit data accumulation result or 16-bit data accumulation result through a preset algorithm to obtain the 8-bit data calculation result or the 16-bit data calculation result, thus completing the frequency domain convolution calculation of mixed precision data;

[0159] The system in this embodiment is based on FPGA and utilizes the characteristics of frequency domain convolution to be compatible with frequency domain convolution calculations for both 8-bit and 16-bit precision data. It can simultaneously process multiple mixed precision data types, effectively improving the utilization efficiency of the DSP in the FPGA and the operation speed of frequency domain convolution.

[0160] The same or similar labels correspond to the same or similar parts;

[0161] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0162] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for accelerating mixed-precision data frequency domain convolution based on FPGA, characterized in that, Includes the following steps: S1: Obtain several 8-bit and 16-bit data; S2: Directly route each 8-bit data to the first storage location for caching to obtain an 8-bit data caching result; divide each 16-bit data equally and route it to the first storage location for caching to obtain a 16-bit data caching result; S3: Select two 8-bit data cache results from the first storage location, pack them into the second storage location for storage, and obtain the 8-bit data storage result; Alternatively, a 16-bit data cache result can be selected from the first storage location and directly input into the second storage location for storage to obtain a 16-bit data storage result; S4: Perform frequency domain convolution calculation on the 8-bit data storage result, and perform offset correction on the frequency domain convolution calculation result of the 8-bit data storage result to obtain the 8-bit data frequency domain convolution result; or perform frequency domain convolution calculation on the 16-bit data storage result to obtain the 16-bit data frequency domain convolution result. The specific method for offset correction of the frequency domain convolution multiplication calculation result of 8-bit data storage result is as follows: The convolution multiplication result of the 8-bit data storage is input into a preset shift register D, and the value of shift register D is determined. The convolution multiplication result is then offset based on the value of shift register D to obtain the 8-bit data frequency domain convolution result, denoted as: Where A and C are data registers, and B is a preset convolution kernel register; The shift register D has a bit width of 48 bits; The specific method for determining the value of shift register D is as follows: The calculation method is controlled using a preset inmode control signal; When the inmode control signal is " "When: If the data in bits 0-7 of data registers A and C are all negative, then the value of shift register D is the inverted value obtained by shifting the data in kernel register B left by 9 bits, denoted as:" ; If the data in bits 0-7 of data registers A and C is negative in one case and non-negative in the other, then the value of shift register D is the inverted value obtained by shifting the data in kernel register B left by 8 bits, denoted as: In other cases, the value of shift register D is 0. When the inmode control signal is " "When: If the data in bits 0-7 of data register A is negative and the data in bits 0-7 of data register C is non-negative, then the value of shift register D is the inverted value obtained by shifting the data in kernel register B left by 8 bits, denoted as:" ; If the data in bits 0-7 of data register A is non-negative and the data in bits 0-7 of data register C is negative, then the value of shift register D is the value obtained by left-shifting the data in kernel register B by 8 bits, denoted as: In other cases, the value of shift register D is 0. S5: Perform an accumulation operation on the frequency domain convolution result of 8-bit data or the frequency domain convolution result of 16-bit data to obtain the accumulated result of 8-bit data or the accumulated result of 16-bit data; S6: The obtained 8-bit or 16-bit data accumulation result is processed by a preset algorithm to recover the data, and the 8-bit or 16-bit data calculation result is obtained, thus completing the frequency domain convolution calculation of mixed precision data.

2. The FPGA-based mixed-precision data frequency domain convolution acceleration method according to claim 1, characterized in that, The specific method for step S2 is as follows: The bit width of the first storage location is 8 bits; For each 8-bit data, it is directly routed to the ordered position in the first storage location for caching, thus obtaining the 8-bit data caching result; For each 16-bit data, divide each 16-bit data into the first 8 bits and the last 8 bits. Then, route the first 8 bits and the last 8 bits to the ordered positions in the first storage location for caching to obtain the 16-bit data caching result.

3. The FPGA-based mixed-precision data frequency domain convolution acceleration method according to claim 2, characterized in that, The specific method for step S3 is as follows: The second storage location has a bit width of 27 bits, including bits 0 to 26; Two 8-bit data cache results are selected from the first storage location and packaged into the second storage location. One 8-bit data cache result is stored in bits 0 to 7 of the second storage location, and the other 8-bit data cache result is stored in bits 18 to 25 of the second storage location. Bit 26 is used as the sign bit. Bits 8 to 17 are filled with zeros to obtain the 8-bit data storage result. Alternatively, select a 16-bit data cache result and directly input it into the second storage location. The first 8 bits and the last 8 bits are stored in bits 0 to 15 of the second storage location, and the 16th bit is used as the sign bit. Bits 17 to 26 are filled with zeros to obtain the 16-bit data storage result.

4. The FPGA-based mixed-precision data frequency domain convolution acceleration method according to claim 3, characterized in that, The specific method for step S4 is as follows: The frequency domain convolution calculation is performed on the 8-bit data storage result. The two 8-bit data storage results are input into data registers A and C respectively to perform convolution multiplication calculations. The convolution multiplication calculation results of the 8-bit data storage result are offset and corrected to obtain the 8-bit data frequency domain convolution result. Alternatively, frequency domain convolution calculation can be performed on the 16-bit data storage result. The two 16-bit data storage results are input into data registers A and C respectively to perform convolution multiplication calculations to obtain the 16-bit data frequency domain convolution result. Store the obtained 8-bit data frequency domain convolution result or 16-bit data frequency domain convolution result into data register E; The data registers A and C have a bit width of 27 bits, and the data register E has a bit width of 48 bits.

5. The FPGA-based mixed-precision data frequency domain convolution acceleration method according to claim 4, characterized in that, The convolution multiplication calculation is specifically as follows: Perform addition or subtraction operations on the two 8-bit or 16-bit data storage results in data registers A and C to obtain the first intermediate result, denoted as: ; Perform a convolution multiplication operation between the first intermediate result and the convolution kernel in the preset convolution kernel register B, denoted as: ; The preset convolution kernel register B has a width of 18 bits and is used to store convolution kernels of 8-bit or 16-bit precision.

6. The FPGA-based mixed-precision data frequency domain convolution acceleration method according to claim 5, characterized in that, The specific method for performing addition or subtraction operations on the two 8-bit or 16-bit data storage results in data registers A and C is as follows: The calculation method is controlled using a preset inmode control signal; When the inmode control signal is " When the condition is met, the two 8-bit or 16-bit data storage results in data registers A and C are added together to obtain the first intermediate result, denoted as: ; When the inmode control signal is " When the result is "", the two 8-bit or 16-bit data storage results in data registers A and C are subtracted to obtain the first intermediate result, denoted as: .

7. The FPGA-based mixed-precision data frequency domain convolution acceleration method according to claim 1, characterized in that, In step S5, the 8-bit data frequency domain convolution result or the 16-bit data frequency domain convolution result is accumulated to obtain an 8-bit data accumulation result or a 16-bit data accumulation result. The specific method is as follows: The 8-bit data frequency domain convolution result or the 16-bit data frequency domain convolution result has a bit width of 16 bits. The obtained 8-bit data frequency domain convolution result is divided equally. The first 8 bits of the 8-bit data frequency domain convolution result are used as the low-order data, and the last 8 bits of the 8-bit data frequency domain convolution result are used as the high-order data. The low-order data and the high-order data are accumulated separately to obtain the 8-bit data accumulation result. The obtained 16-bit data frequency domain convolution results are directly accumulated to obtain the 16-bit data accumulation result.

8. The FPGA-based mixed-precision data frequency domain convolution acceleration method according to claim 1, characterized in that, In step S6, the obtained 8-bit data accumulation result or 16-bit data accumulation result is processed by a preset algorithm to recover the data, thereby obtaining the 8-bit data calculation result or the 16-bit data calculation result. The specific method is as follows: The obtained 8-bit or 16-bit data accumulation results are processed by a preset algorithm to recover the data. The real and imaginary parts of the 8-bit or 16-bit data accumulation results are calculated respectively to obtain the complex number form of the 8-bit or 16-bit data calculation results.

9. A mixed-precision data frequency domain convolution acceleration system based on FPGA, employing any one of the mixed-precision data frequency domain convolution acceleration methods based on FPGA according to claims 1 to 8, characterized in that, include: Data acquisition module: used to acquire several 8-bit and 16-bit data; Data routing module: used to directly route each 8-bit data to the first storage location for caching, obtaining an 8-bit data caching result; and to divide each 16-bit data equally and route it to the first storage location for caching, obtaining a 16-bit data caching result. Data preprocessing module: used to select two 8-bit data cache results from the first storage location, package them into the second storage location for storage, and obtain 8-bit data storage results; Alternatively, a 16-bit data cache result can be selected from the first storage location and directly input into the second storage location for storage to obtain a 16-bit data storage result; Frequency domain convolution module: Used to perform frequency domain convolution calculations on 8-bit data storage results, and to perform offset correction on the frequency domain convolution calculation results of the 8-bit data storage results to obtain the 8-bit data frequency domain convolution result; or to perform frequency domain convolution calculations on 16-bit data storage results to obtain the 16-bit data frequency domain convolution result; the specific method for offset correction on the frequency domain convolution multiplication calculation results of 8-bit data storage results is as follows: The convolution multiplication result of the 8-bit data storage is input into a preset shift register D, and the value of shift register D is determined. The convolution multiplication result is then offset based on the value of shift register D to obtain the 8-bit data frequency domain convolution result, denoted as: Where A and C are data registers, and B is a preset convolution kernel register; The shift register D has a bit width of 48 bits; The specific method for determining the value of shift register D is as follows: The calculation method is controlled using a preset inmode control signal; When the inmode control signal is " "When: If the data in bits 0-7 of data registers A and C are all negative, then the value of shift register D is the inverted value obtained by shifting the data in kernel register B left by 9 bits, denoted as:" ; If the data in bits 0-7 of data registers A and C is negative in one case and non-negative in the other, then the value of shift register D is the inverted value obtained by shifting the data in kernel register B left by 8 bits, denoted as: In other cases, the value of shift register D is 0. When the inmode control signal is " "When: If the data in bits 0-7 of data register A is negative and the data in bits 0-7 of data register C is non-negative, then the value of shift register D is the inverted value obtained by shifting the data in kernel register B left by 8 bits, denoted as:" ; If the data in bits 0-7 of data register A is non-negative and the data in bits 0-7 of data register C is negative, then the value of shift register D is the value obtained by left-shifting the data in kernel register B by 8 bits, denoted as: In other cases, the value of shift register D is 0. Data accumulation module: used to accumulate the frequency domain convolution results of 8-bit data or 16-bit data to obtain the accumulated 8-bit data result or the accumulated 16-bit data result; Data recovery module: Used to perform data recovery processing on the obtained 8-bit data accumulation result or 16-bit data accumulation result through a preset algorithm to obtain the 8-bit data calculation result or 16-bit data calculation result, and complete the frequency domain convolution calculation of mixed precision data.

Images