CNN inference architecture design method based on real FFT and IFFT fast convolution
By using real FFT and IFFT fast convolution methods in CNN inference processor, the problem of high computational complexity of convolution layer is solved, and the CNN inference architecture with low complex computing resources is realized, which improves the computing efficiency and real-timeness. It is suitable for A distributed unmanned system with limited computing resources.
Patent Information
- Application Number
- CN202411024760.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-29
AI Technical Summary
Existing CNN inference processors have high complex computing resource consumption in convolutional layer computing, resulting in low computing efficiency and large power consumption, making it difficult to implement real-time inference in distributed unmanned systems with limited computing resources.
Using a fast convolution method based on real numbers FFT and IFFT, the two-dimensional real numbers FFT transformation and point multiplication calculation is performed by dividing feature data into small blocks, reducing the calculation complexity, and using double-port cache and a single real number multiplier to realize complex multiplication. Reduce computing resource requirements.
It improves the computing efficiency and real-time nature of CNN inference, reduces computing resource consumption, and realizes a CNN inference architecture with low complexity and fast computing, which is suitable for end-load intelligent applications with limited computing resources.
Smart Images

Figure CN118551850B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of neural network technology, and in particular to a CNN inference architecture design method based on real number FFT and IFFT fast convolution. Background Art
[0002] Deep learning inference processors are the carriers that promote the intelligent practical application of distributed unmanned systems. Among them, convolutional neural network (CNN) inference processing is a typical representative, which can support applications such as situational awareness, guidance tracking, and target detection. Lightweight CNN inference must not only reduce the consumption of processor computing resources and solve the constraints of power consumption and volume of computing equipment in distributed unmanned systems; it must also ensure real-time computing, improve the efficiency of online processing of dense data, and quickly calculate the inference results to reduce the load of invalid data return.
[0003] Therefore, it is urgent to carry out research on low-complexity and low-cost CNN fast reasoning methods and architectures for end-load intelligent applications with limited computing resources. When stripping the classic target recognition network reasoning model, nearly 80% of the calculations are concentrated in the convolution layer. Therefore, the complexity of the convolution layer calculation is the key to affecting the CNN reasoning efficiency and resource usage.
[0004] The convolution layer mainly performs convolution operations by sliding the kernel function on the feature map for filtering. Its essence is time-domain filtering, which can reduce the computational complexity (required computational cycle) by transforming the computational domain to improve the real-time and efficiency of convolution calculations, thereby improving the real-time and efficiency of CNN reasoning. Typical methods include acceleration methods based on FFT and Winograd transform.
[0005] After the CNN inference operation introduces FFT convolution, the related operations become complex number calculations, and complex domain operations will lead to additional storage and computing resources. Generally speaking, complex number storage is twice that of real number storage, and the number of multipliers required for complex number multiplication is four times that of real number multiplication. Summary of the invention
[0006] Based on this, it is necessary to provide a CNN inference architecture design method based on real FFT and IFFT fast convolution to address the above technical problems. This method uses simple real FFT transform to replace complex complex FFT transform calculation, thereby improving the overall CNN inference efficiency, improving the CNN inference architecture, and reducing.
[0007] A CNN inference architecture design method based on real number FFT and IFFT fast convolution, the method comprising:
[0008] The real number feature data is divided into several overlapping small blocks, and all the block feature data are processed online using a two-dimensional real number FFT transformation calculation module to obtain block frequency domain feature data.
[0009] A dual-port cache + single real number multiplier is used to calculate the point product of the block frequency domain feature data and the block frequency domain kernel function to obtain a block frequency domain point product result; wherein, slow cache data is used to quickly read data during the point multiplication process, and a single real number multiplier is used to serially calculate the real and imaginary parts of the complex multiplication; the block frequency domain kernel function is obtained by pre-calculating the two-dimensional FFT transformation of the real kernel function after zero padding through offline computing resources and storing the correlation coefficient.
[0010] The two-dimensional real number IFFT calculation module is used to process the block frequency domain point multiplication results to obtain the time domain feature data.
[0011] In one embodiment, a two-dimensional real number IFFT calculation module is used to process the block frequency domain dot multiplication result to obtain time domain feature data, including:
[0012] When executing multi-channel parallel convolution calculations in the convolution layer, the block frequency domain point multiplication results of all channels are accumulated and then processed using a two-dimensional real number IFFT calculation module to obtain time domain feature data.
[0013] In one embodiment, a dual-port buffer + single real number multiplier is used to calculate the dot product of the block frequency domain feature data and the block frequency domain kernel function to obtain the block frequency domain dot product result, including:
[0014] When performing multi-channel parallel convolution calculations within the convolution layer, a shared two-dimensional FFT transform is used to perform point multiplication on the block frequency domain feature data and the kernel functions of the FFT transforms of different channels; the block frequency domain point multiplication result of each channel is obtained.
[0015] In one embodiment, when calculating the complex FFT transform, the two-dimensional real number FFT transform calculation module separates the real part and the imaginary part and uses two consecutive one-dimensional real number FFT transforms to perform calculations respectively.
[0016] In one embodiment, in a two-dimensional real number FFT transformation calculation module, a double-port buffer is used for the result after the first one-dimensional real number FFT transformation, and the real part and the imaginary part of the first M / 2+1 complex data after the FFT transformation are retained; a double-port buffer is used for the result after the second one-dimensional real number FFT transformation, and the real part and the imaginary part of the first M×(M / 2+1) complex data after the FFT transformation are retained;
[0017] In one embodiment, when calculating the complex IFFT transform, the two-dimensional real IFFT calculation module separates the real part and the imaginary part and performs calculations respectively using two consecutive one-dimensional real IFFT transforms.
[0018] In one of the embodiments, in the two-dimensional real number IFFT calculation module, the result after the first one-dimensional real number IFFT transformation is cached, and the real and imaginary parts of the first M×(M / 2+1) complex data are retained; the result after the second one-dimensional real number IFFT transformation is cached, and the real and imaginary parts of the first M / 2+1 complex data are retained; where M is the number of points of the FFT transformation and the IFFT transformation.
[0019] In one of the embodiments, a quadratic value is used as a point value for FFT transformation and IFFT transformation, and the point value is no greater than 8.
[0020] In one embodiment, a short-point real number FFT transform is used, and the rotation factors in the butterfly operation include: 0, ±1 and ; Among them, the multiplication with a rotation factor of 1 is implemented using an adder-subtractor; the rotation factor is The multiplication is implemented by arithmetic left shift followed by accumulation or by constant multiplier IP core.
[0021] In one of the embodiments, in the two-dimensional real number FFT transform calculation module, the two-dimensional FFT transform is implemented by a real number FFT transform of rows first and columns later, or by a real number FFT transform of columns first and rows later; in the two-dimensional real number IFFT calculation module, the two-dimensional IFFT transform is implemented by a real number IFFT transform of columns first and rows later, or by a real number IFFT transform of rows first and columns later.
[0022] The above-mentioned CNN reasoning architecture design method based on real FFT and IFFT fast convolution reduces the computational complexity by transforming the computational domain of convolution to improve the real-time and efficiency of convolution calculation, thereby improving the computational real-time and efficiency of the CNN reasoning process; adopts real FFT transformation and IFFT to realize complex complex FFT transformation and IFFT, and adopts short point FFT to reduce the resource consumption of rotation factor multiplication; by utilizing the linear characteristics of convolution, merges and shares the input two-dimensional FFT and output IFFT, further reducing the computational resources required for transformation calculation; adopts serial real multiplication to realize complex point multiplication. This method realizes a CNN reasoning architecture with low computational resource cost based on low-complexity fast calculation. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 It is a flowchart of a method for designing a CNN reasoning architecture based on real FFT and IFFT fast convolution in one embodiment;
[0024] Figure 2 It is a schematic diagram showing that the result of two-dimensional FFT calculation of kernel function and characteristic data block presents central conjugate symmetry when the input is a real number in another embodiment;
[0025] Figure 3 A schematic diagram of value distribution after offline FFT calculation when M=8 in another embodiment;
[0026] Figure 4 A schematic diagram of a storage method of values after offline FFT calculation when M=8 in another embodiment;
[0027] Figure 5 A schematic diagram of storage operations when a complex number FFT transform is implemented by splitting a complex number into separate real and imaginary parts and then separately calculating the real and imaginary parts for real number FFT transform in another embodiment;
[0028] Figure 6 The figure is a schematic diagram of the principle diagram and functional implementation method of completing fast convolution calculation based on FFT in another embodiment, wherein (a) is a schematic diagram of the calculation process of the multi-channel convolution layer, and (b) is a schematic diagram of the specific implementation of the calculation;
[0029] Figure 7 A schematic diagram of a multi-channel input and multi-channel output convolution calculation method in another embodiment;
[0030] Figure 8 Schematic diagram of calculation steps for implementing two-dimensional FFT through two real number FFT transformations in another embodiment, wherein (a) is a theoretical calculation step, and (b) is a calculation step of the present application;
[0031] Fig. 9 A schematic diagram of a calculation method for implementing complex FFT transformation by using real FFT transformation for real and imaginary parts respectively in another embodiment;
[0032] Fig.10 A schematic diagram of a flow chart of implementing complex multiplication by driving a single real multiplier through a dual-port buffer slow clock accessing a fast clock in another embodiment;
[0033] Fig.11 A schematic diagram of another embodiment of a method of implementing complex multiplication by using a dual-port buffer slow clock to access a fast clock and drive a single real multiplier;
[0034] Fig.12 is the mean square error of the calculation result based on Winograd fast convolution transform in another embodiment;
[0035] Fig.13 is the maximum absolute error of the calculation result based on the Winograd fast convolution transform in another embodiment. DETAILED DESCRIPTION
[0036] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0037] The traditional convolution calculation method based on time domain sliding filtering in the CNN reasoning architecture is transformed to the frequency domain through two-dimensional FFT to reduce the complexity of the overall convolution calculation. Since transforming the calculation domain is essentially a trade-off between resources and efficiency, the FFT-based convolution acceleration method involves dense complex FFT transformations, data point multiplications, etc., and the consumed computing resources are inconsistent with the lightweight requirements of end-to-end intelligent reasoning. For N×N dimensional data input and K×K dimensional convolution kernel, the number of calculations required for the convolution operation using traditional sliding filtering at a unit step is , this indicator can be equivalent to the calculation cycle of a single layer of convolution. Based on the FFT fast convolution operation architecture, through the feature map block and transformation, the unit step in the time domain convolution increases to M-K+1 after the transformation domain, and the number of calculations required is , where M is the number of points of FFT transformation, and is generally greater than K. It can be seen that the number of calculations based on FFT fast convolution method is less than that of sliding convolution. By reducing the convolution calculation cycle through FFT transformation, fast convolution is achieved, thereby accelerating the overall reasoning.
[0038] In one embodiment, Figure 1 As shown, a CNN reasoning architecture design method based on real FFT and IFFT fast convolution is provided, and the method includes the following steps:
[0039] Step 100: Divide the real number feature data into a number of overlapping small blocks, and process all the block feature data online using a two-dimensional real number FFT transformation calculation module to obtain block frequency domain feature data.
[0040] Specifically, the real number feature data is divided into a number of overlapping small blocks to obtain a number of block feature data, and the size of the block feature data is equal to the number of FFT transformation and IFFT transformation points.
[0041] The two-dimensional real FFT transform calculation is used to replace the two-dimensional complex FFT transform calculation, thereby reducing the computing resources required for fast convolution.
[0042] The N×N real feature data is divided into several overlapping small blocks, each block feature data has a dimension of M×M, and the block feature data is calculated online by two-dimensional FFT to obtain block frequency domain feature data. The two-dimensional FFT transform is completed by two consecutive one-dimensional real FFT transforms, and the row and column block feature data are calculated respectively; the input of the first one-dimensional FFT transform is the feature real data; the input of the second one-dimensional FFT transform is the complex data, and the second one-dimensional FFT transform completes the complex FFT transform by splitting the real and imaginary parts of the complex data using two consecutive real FFT transforms.
[0043] For convolutional layers containing multiple channels, all channels can be transformed by sharing the two-dimensional FFT.
[0044] Online refers to using FPGA resources to perform real-time calculations on data streams.
[0045] Step 102: Use a dual-port cache + single real number multiplier to calculate the point product of the block frequency domain feature data and the block frequency domain kernel function to obtain the block frequency domain point product result; wherein, in the point multiplication process, slow cache data is used to quickly read data, and a single real number multiplier is used to serially calculate the real and imaginary parts of the complex multiplication; the block frequency domain kernel function is obtained by pre-calculating the two-dimensional FFT transformation of the real kernel function after zero padding through offline computing resources and storing the correlation coefficient.
[0046] Specifically, the size of the kernel function after zero padding is equal to the number of FFT transform points and IFFT transform points.
[0047] The block frequency domain kernel function is obtained by pre-calculating the two-dimensional FFT transformation and storing the correlation coefficients through a computer or other computing resources; according to the central conjugate symmetry characteristics after the two-dimensional FFT transformation, only half of the data is retained.
[0048] The data size of the block frequency domain kernel function is generally 3×3~7×7, and the FFT transform generally uses quadratic power. Therefore, when calculating the kernel function FFT transform, the kernel function needs to be filled with zeros and expanded to M points. Generally, the two-dimensional FFT value of the kernel function is calculated offline and the correlation coefficient is stored in the storage; according to the central conjugate symmetry characteristics, , in fact, only the real and imaginary parts of the first M×(M / 2+1) data of the two-dimensional FFT data need to be stored; among them, is the Fourier transform.
[0049] The result of two-dimensional FFT calculation of kernel function and characteristic data block is central conjugate symmetric. When the input is real number, the result of two-dimensional FFT calculation of kernel function and characteristic data block is central conjugate symmetric. Figure 2 As shown. Central conjugation means that the matrix has conjugate symmetry relative to the midpoint.
[0050] The K×K convolution kernel is expanded to M×M by filling zeros. When M=8, the numerical distribution after offline FFT transformation is shown as follows: Figure 3 shown. Figure 3 R(u,v) is the real part, I(u,v) is the imaginary part, and u and v are the indices of the two-dimensional position.
[0051] The K×K convolution kernel is expanded to M×M by filling zeros. When M=8, the numerical storage method after offline FFT transformation is as follows: Figure 4 As shown. Using the central conjugate symmetry characteristics, the real and imaginary coefficients are extracted and stored, and its dimension is M×M.
[0052] The two-dimensional real number FFT transformation calculation module uses a dual-port buffer to retain the calculation result after the first one-dimensional FFT transformation after the first one-dimensional FFT transformation.
[0053] The dot multiplication of the block frequency domain feature data and the block frequency domain kernel function is realized by using a cache + fast clock driven real multiplier. The cache is written with a slow clock, and the cache is read with a fast clock. A single real multiplier is used to serially calculate the real and imaginary parts required for complex multiplication. The complex dot multiplication is converted into a serial fast clock real dot multiplication, which reduces the computing resource requirements.
[0054] Complex point multiplication is implemented by using a cache + a real multiplier. FFT(col1(k))=A k +j×B k , which can be realized by FFT transformation of the real and imaginary parts of col1(k) and then accumulating them. It is also calculated synchronously in each result operation cycle of the imaginary part FFT (col1(k)). The corresponding kernel function point multiplication value is a complex number (which is expressed as: f 1,k +j×g 1,k , where f 1,k is the real part, g 1,k is the imaginary part); where col1(k) is the complex data in the kth column, A k , B k are the real and imaginary parts of FFT(col1(k)), respectively, and j is a complex unit.
[0055] In order to complete the complex multiplication and distinguish the real and imaginary parts within one calculation cycle, a fast clock-driven real multiplier is used to serially calculate the real and imaginary parts of the dot product. The schematic diagram of the storage operation when the complex FFT transform is realized by splitting the complex number into separate real and imaginary parts and then calculating the real and imaginary parts separately is shown in the figure. Figure 5 shown.
[0056] Step 104: Process the block frequency domain dot product result using a two-dimensional real number IFFT calculation module to obtain time domain feature data.
[0057] Specifically, the block frequency domain dot product result obtained by dot multiplication of the block frequency domain feature data and the block frequency domain kernel function is subjected to a two-dimensional IFFT transformation, which is then transformed into the time domain and then subjected to conventional calculations such as pooling, activation, and fully connected layers.
[0058] The two-dimensional real number IFFT calculation module uses a dual-port buffer to retain the calculation result after the first one-dimensional IFFT transformation after the first one-dimensional IFFT transformation.
[0059] The two-dimensional real IFFT transform is used to replace the two-dimensional complex IFFT transform, thereby reducing the computational resources required for fast convolution.
[0060] After two-dimensional real number FFT transformation and dot multiplication, it still has the central conjugate characteristic, so only the first half of the central conjugate symmetric complex data needs to be calculated.
[0061] Take the multi-channel convolution operation in a CNN inference architecture designed using this method as an example. Figure 6 As shown. For the convolution layer of N×N feature data, K×K convolution kernel and C1 input channel, M-point FFT transform and IFFT transform are used, the feature data block step is M-K+1, and the dimension of the output data after convolution of each channel is R×R, where R=N-K+1. The K×K convolution kernel is filled with zeros and expanded to M×M. The correlation coefficient is calculated by offline FFT, and the real and imaginary coefficients are extracted by using the central conjugate symmetry characteristics and then put into the storage space, whose dimension is M×M. Multiple channels are finally accumulated into an R×R-dimensional output data, and subsequent operations such as pooling activation are performed. The above operations are consistent on the remaining output channels, so the convolution data of C2 channels is output. Figure 6 (a) is a schematic diagram of the calculation process of the multi-channel convolutional layer. Figure 6 (b) is a schematic diagram of the specific implementation of the calculation.
[0062] In the above-mentioned CNN reasoning architecture design method based on real FFT and IFFT fast convolution, the method reduces the computational complexity (required computational cycle) by transforming the computational domain of convolution to improve the real-time and efficiency of convolution calculation, thereby improving the computational real-time and efficiency of the CNN reasoning process; adopts real FFT transformation and IFFT to realize complex complex FFT transformation and IFFT, and adopts short-point FFT transformation to reduce the resource consumption of rotation factor multiplication; utilizes the linear characteristics of convolution to merge and share the input two-dimensional FFT and output IFFT to further reduce the computational resources required for transformation calculation; adopts serial real multiplication to realize complex point multiplication. This method realizes a CNN reasoning architecture with low computational resource cost based on low-complexity fast calculation.
[0063] In one embodiment, step 104 includes: when performing multi-channel parallel convolution calculations in the convolution layer, accumulating the block frequency domain point multiplication results of all channels and then processing them using a two-dimensional real number IFFT calculation module to obtain time domain feature data.
[0064] Specifically, for the multi-channel convolution layer, each channel can be individually IFFT transformed and accumulated after calculating the dot product, but this approach requires that each channel perform IFFT calculation, resulting in a large number of IFFT calculation modules. Therefore, according to the linear characteristics of the IFFT transform, the present application accumulates the frequency domain dot product results of different channels for the multi-channel parallel convolution calculation in the convolution layer, and then performs a two-dimensional IFFT transform; the two-dimensional IFFT transform uses two consecutive one-dimensional real IFFT transforms to calculate the rows and columns of the dot product data respectively. Among them, the first one-dimensional complex IFFT transform is implemented by splitting the real and imaginary parts and using the real IFFT transform; the second one-dimensional IFFT transform has a conjugate symmetric input. The values after the dot product of all channels are accumulated first, and only one two-dimensional IFFT calculation module is required when performing the two-dimensional IFFT transform, which can reduce the overall number of IFFT calculation modules.
[0065] In one embodiment, step 102 includes: when performing multi-channel parallel convolution calculations within the convolution layer, using a shared two-dimensional FFT transform method to perform point multiplication on the block frequency domain feature data and the kernel function of the FFT transform of different channels; and obtaining the block frequency domain point multiplication result of each channel.
[0066] Specifically, for multi-channel convolution layers, FFT transformation can be performed separately and point multiplication can be performed with the kernel function after FFT transformation, but this approach requires that each channel be FFT transformed; therefore, according to the linear characteristics of FFT transformation, this application adopts a shared two-dimensional FFT transformation method for multi-channel parallel convolution calculations in the convolution layer, and first performs two-dimensional FFT transformation on all channel feature data blocks, and then uses the calculated results and the kernel function of different channel FFT transformations for point multiplication. At this time, only one set of two-dimensional FFT calculation modules is needed, which can reduce the number of FFT calculation modules overall.
[0067] After the shared two-dimensional FFT transformation, point multiplication is performed according to independent channels respectively. After accumulating the calculation results of different channels, IFFT transformation is performed again, and pooling and activation operations are performed on the convolution results.
[0068] The convolution layer combines and shares the FFT transform and IFFT transform calculation modules to implement a multi-channel input and multi-channel output convolution calculation method. Figure 7 shown.
[0069] In one embodiment, when calculating the complex FFT transform, the two-dimensional real number FFT transform calculation module separates the real part and the imaginary part and uses two consecutive one-dimensional real number FFT transforms to perform calculations respectively.
[0070] Specifically, when calculating the complex FFT transform, the real part and the imaginary part are calculated separately, that is, FFT(x[k])=FFT(r[k]+j×im[k])=FFT(r[k])+j×FFT(im[k]), and the complex complex calculation is split into two consecutive real FFT transforms; where x[k] is the kth complex array, r[k] and im[k] are the real part and imaginary part of x[k], respectively.
[0071] The schematic diagram of the calculation steps of the two-dimensional FFT through two one-dimensional real number FFT transformations is as follows Figure 8 As shown, (a) is the theoretical calculation step, and (b) is the calculation step of this application. By splitting the complex number into separate real and imaginary parts, two real FFT transforms are used to calculate the real part and the imaginary part respectively. The schematic diagram of the calculation method of using real FFT transform to realize complex FFT transform for the real and imaginary parts is shown in Fig. 9 As shown. Since the feature input of the convolutional layer is real, the first one-dimensional row FFT transform is a real FFT transform, and its output is conjugate symmetric, that is, (k=0,…M-1), only keep (k=0,…M / 2) The real and imaginary parts of the numerical part are stored row by row in the buffer Mem(u,v).
[0072] For column FFT, according to the current method, Fig. 9 In the example, FFT(col(k)) is used, where col(k) is the kth column of the complex array, col(k)=col_real(k)+j×col_imag(k), col_real(k) and col_imag(k) are the real and imaginary parts of col(k) respectively. The essence of the method is complex FFT transformation. However, the real and imaginary parts are split into two parts, FFT(col(k))=FFT(col_real(k))+j×FFT(col_imag(k)), that is, the real and imaginary parts are calculated separately, and then the original complex output FFT(col(k)) is formed. The simplified real FFT transformation is used to replace the complex complex FFT transformation to achieve the purpose.
[0073] In one of the embodiments, in the two-dimensional real number FFT transformation calculation module, a dual-port cache is used for the result after the first one-dimensional real number FFT transformation, and the real and imaginary parts of the first M / 2+1 complex data after the FFT transformation are retained; a dual-port cache is used for the result after the second one-dimensional real number FFT transformation, and the real and imaginary parts of the first M×(M / 2+1) complex data after the FFT transformation are retained.
[0074] Specifically, in the two-dimensional real number FFT transform calculation module, the result of the first one-dimensional real number FFT transform calculation is a complex number, but because the calculation result is conjugate symmetric, the cache only retains the real and imaginary parts of the first M / 2+1 complex output results (M is the number of FFT transform points). After the second one-dimensional real number FFT transform, because the two-dimensional FFT transform is central conjugate symmetric, a dual-port cache is used to retain the real and imaginary parts of the first M×(M / 2+1) complex data after the FFT transform.
[0075] In one embodiment, when calculating the complex IFFT transform, the two-dimensional real IFFT calculation module separates the real part and the imaginary part and performs calculations respectively using two consecutive one-dimensional real IFFT transforms.
[0076] Specifically, the two-dimensional real IFFT calculation module calculates the real part and the imaginary part separately when calculating the complex IFFT transform, that is, IFFT(x[k])= IFFT(r[k]+j×im[k])= IFFT(r[k])+j×IFFT(im[k]). The complex complex calculation can be divided into two consecutive real IFFT transforms.
[0077] For the two-dimensional IFFT transform, two one-dimensional IFFT transforms are used to complete it; the first one-dimensional IFFT transform input is complex data, which can decompose the real and imaginary parts of the complex data, and complete the complex IFFT transform through two consecutive real IFFT transforms; the second one-dimensional IFFT transform input is also complex, but has conjugate symmetry characteristics, and its output is still real.
[0078] In one of the embodiments, in the two-dimensional real number IFFT calculation module, the result after the first one-dimensional real number IFFT transformation is cached, and the real and imaginary parts of the first M×(M / 2+1) complex data are retained; the result after the second one-dimensional real number IFFT transformation is cached, and the real and imaginary parts of the first M / 2+1 complex data are retained; where M is the number of points of the FFT transformation and the IFFT transformation.
[0079] Specifically, in the two-dimensional real IFFT calculation module, the input of the first one-dimensional real IFFT transformation is a complex number, the real part and the imaginary part are calculated separately, and then the cache retains only the real part and the imaginary part of the first M×(M / 2+1) complex data; the input of the second one-dimensional real IFFT transformation is a complex number, the real part and the imaginary part are calculated separately, and then the cache retains the real part and the imaginary part of the first M / 2+1 complex data (according to the relevant properties, the input is also conjugate symmetric, so only the real part and the imaginary part of the first M / 2+1 complex data are needed); where M is the number of points of FFT transformation and IFFT transformation.
[0080] For the second one-dimensional real FFT transform, since the two-dimensional FFT transform is central conjugate symmetric, the buffer only retains the real and imaginary parts of the first M×(M / 2+1) complex data.
[0081] In one of the embodiments, a quadratic value is used as a point value for FFT transformation and IFFT transformation, and the point value is no greater than 8.
[0082] Specifically, a quadratic value is used as the point value of the FFT transformation and the IFFT transformation, and is not greater than 8. Preferably, a short point number such as 4 or 8 is used as the point value of the FFT transformation and the IFFT transformation.
[0083] By using short-point real number FFT transform and IFFT transform, the multiplication implementation structure is simplified by sharing the internal rotation factor multiplication, and the calculation resources of FFT transform and IFFT transform are reduced.
[0084] In one embodiment, a short-point real FFT transform is used, and the rotation factors in the butterfly operation are only 0, ±1 and ; Among them, the multiplication with a rotation factor of 1 is implemented using an adder-subtractor; the rotation factor is The multiplication is implemented by arithmetic left shift followed by accumulation or by constant multiplier IP core.
[0085] Specifically, when using M=8 short-point FFT and IFFT, according to the characteristics of butterfly operation, the rotation factor multiplication is only the following data, namely 0, ±1 and ; Among them, the multiplication with a rotation factor of 1 is implemented using an adder-subtractor; the rotation factor is Multiplication is implemented by arithmetic logic left shift followed by accumulation or by a constant multiplier IP core.
[0086] In one embodiment, in a two-dimensional real number FFT transform calculation module, the two-dimensional FFT transform is implemented by a real number FFT transform of rows first and columns second, or by a real number FFT transform of columns first and rows second. In a two-dimensional real number IFFT calculation module, the two-dimensional IFFT transform is implemented by a real number IFFT transform of columns first and rows second, or by a real number IFFT transform of rows first and columns second.
[0087] Specifically, the first one-dimensional real IFFT transform, whose input is a complex number, separates the real part and the imaginary part, calculates the real part and the imaginary part respectively, and only retains the real part and the imaginary part of the first M×(M / 2+1) complex data in the cache.
[0088] In a verification embodiment, the current mainstream CNN reasoning acceleration method, that is, the CNN reasoning architecture design based on Winograd fast transformation, is compared. For the Lenet-5 neural network, the process design based on real FFT convolution calculation is completed, and the accuracy and resource consumption based on Winograd (Wino) fast convolution are compared. The accuracy models of the two are implemented based on Matlab, and the hardware is implemented based on Verilog. M=8 point FFT transformation and 3×3 dimensional convolution kernel function are selected, and Wino uses F(6×6,3×3) transformation coefficient matrix. Such a configuration ensures that the output size of FFT and Wino is R=M-K+1=6, which can ensure the consistency of modules such as feature map splitting and overlapping combination, and only replaces the convolution calculation module. Among them, the FFT transformation uses the basic radix-2 butterfly operation, and uses logical left shift to realize the multiplication of non-0 and 1 rotation factors, so there is no DSP module (no multiplier) during logic synthesis. Constant coefficient multiplication in the Wino algorithm is also implemented by logical left and right accumulation. Design relevant process controllers to complete storage read and write control of feature map input, convolution kernel input, and overall FFT transform and IFFT transform calculation process control.
[0089] First, the data accuracy requirements for high-precision convolution based on FFT transform and Wino algorithm are analyzed in Matlab. The reason for limiting the application of FFT transform to mainstream CNN inference network such as 3×3~5×5 small-size kernel functions is that the relevant calculations are in the complex domain. This problem is solved by this method, so we start with accuracy analysis and compare the data accuracy based on FFT transform and Wino. The first step is to determine the data accuracy required for the key module point multiplication in Wino transform, considering that the point multiplication is a fixed-point multiplication, while other transforms / inverse transforms maintain double precision. Compare the traditional convolution implemented with double precision for the basic reference group object.
[0090] The flowchart of implementing complex multiplication by using a dual-port buffer with a slow clock accessing a fast clock to drive a single real multiplier is shown in the figure. Fig.10 As shown in the figure, a dual-port buffer with slow clock accesses fast clock to drive a single real multiplier to realize complex multiplication. Fig.11 As shown. Fig.10 , Fig.11 It can be seen that in the Wino transform fast convolution calculation, the decimal point accuracy of the kernel function involved in the point multiplication greatly affects the accuracy of the convolution layer. According to the Wino inverse transform matrix configured with F(6×6,3×3), the coefficient with the largest absolute value is ±32
[13] . Such a large coefficient will amplify the calculation error of the point multiplication part, so a higher-precision point multiplication is required, that is, a wider word length is required to complete the point multiplication. Compared with the FFT transform, its coefficient is basically maintained at [-1,1], and the point multiplication accuracy does not need to be too high.
[0091] Based on the above conclusions, the accuracy characteristics of the first convolution layer of Lenet-5 are analyzed according to the configuration shown in Table 1. The traditional temporal sliding window convolution form is used as the Golden Reference accuracy reference value for error analysis, and the data accuracy is double-precision floating point.
[0092] Table 1 Configuration parameters list
[0093]
[0094] The convolution calculation based on FFT transform has much better accuracy than the fast convolution calculation based on Wino transform, but the required data word length and data accuracy are much lower than those based on Wino transform, which means that the required computing resources are much lower than those based on Wino transform.
[0095] In summary, for the Winograd inverse transformation matrix, the absolute value of the maximum coefficient is far greater than 1, the point multiplication error will be coupled in the inverse transformation, and in the worst case it will grow exponentially. This will inevitably require that the data accuracy in the Winograd point multiplication calculation be higher than the data point multiplication based on the FFT transformation. The absolute value of the maximum coefficient in the Winograd transformation matrix is far greater than 1, which means that the increase in data word length during the transformation process far exceeds the increase in word length based on the FFT transformation.
[0096] In another verification embodiment, in order to more accurately analyze the computing resources of the two, a Verilog functional model is then established and the convolution layer resources (including the process controller) are estimated and compared in Vivado.
[0097] The multipliers and adders used in both are implemented using LUTs. According to the comprehensive results, the number of LUTs required for Wino-based parallel convolution design is 10464, and the FFT-based design saves nearly 30% of resource consumption compared to the Wino-based design.
[0098] For the calculation of the first convolution layer of the Lenet-5 convolutional neural network, the accuracy and data accuracy of the Winograd-based and FFT-based fast convolution algorithms are compared. Specifically, for the first convolution layer of the Lenet-5 neural network, the calculation accuracy of the Winograd fast convolution transform is analyzed. The variable conditions are the word length and the number of decimal places after the kernel function in the point multiplication is transformed, and the accuracy indicator is the mean square error. The mean square error of the calculation result based on the Winograd fast convolution transform is as follows: Fig.12 As shown in the figure. For the first convolution layer of the Lenet-5 neural network, based on Winograd fast convolution calculation accuracy analysis, the variable conditions are the word length and decimal point of the kernel function after the point multiplication, and the accuracy index is the maximum absolute error. The maximum absolute error of the calculation result based on Winograd fast convolution transformation is shown in the figure. Fig.13 shown.
[0099] In another verification embodiment, for the calculation of the first convolution layer of the Lenet-5 convolutional neural network, the resource consumption of the Winograd-based and FFT-based fast convolution algorithms is compared.
[0100] The characteristics of the method of the present application include: using real number FFT transformation and IFFT transformation to replace the previous complex number FFT transformation and IFFT transformation, reducing; using short point number FFT transformation and IFFT transformation, limiting the rotation factors therein to 0, ±1 and , the multiplication of the rotation factor 0 / 1 is realized by the adder-subtractor. The multiplication can be implemented by arithmetic left shift followed by accumulation or constant multiplier IP core; the frequency domain complex point multiplication of block data is realized by using cache + fast clock driven real number multiplier; for multi-channel input and output, the merged shared two-dimensional FFT transform and IFFT transform method is adopted to further reduce the resource consumption caused by the transform.
[0101] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0102] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0103] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A CNN inference architecture design method based on real FFT and IFFT fast convolution, It is characterized in that The method comprises: The real number feature data is divided into several overlapping small blocks, and all the block feature data are processed online using a two-dimensional real number FFT transformation calculation module to obtain block frequency domain feature data; A dual-port cache + single real number multiplier is used to calculate the point product of the block frequency domain feature data and the block frequency domain kernel function to obtain a block frequency domain point product result; wherein, during the point product process, slow cache data is used to quickly read data, and a single real number multiplier is used to serially calculate the real part and imaginary part of the complex multiplication; the block frequency domain kernel function is obtained by pre-calculating the two-dimensional FFT transformation of the real number kernel function after zero padding through offline computing resources and storing the correlation coefficient; According to the block frequency domain dot multiplication result, a two-dimensional real number IFFT calculation module is used to process to obtain time domain feature data; Wherein, when calculating the complex FFT transform, the two-dimensional real number FFT transform calculation module separates the real part and the imaginary part and respectively calculates them using two consecutive one-dimensional real number FFT transforms; in the two-dimensional real number FFT transform calculation module, a dual-port cache is used for the result after the first one-dimensional real number FFT transform, and the real part and the imaginary part of the first M / 2+1 complex data after the FFT transform are retained; a dual-port cache is used for the result after the second one-dimensional real number FFT transform, and the real part and the imaginary part of the first M×(M / 2+1) complex data after the FFT transform are retained; When calculating the complex IFFT transform, the two-dimensional real number IFFT calculation module separates the real part and the imaginary part and uses two consecutive one-dimensional real number IFFT transforms to perform calculations respectively; in the two-dimensional real number IFFT calculation module, the result after the first one-dimensional real number IFFT transform is cached, and the real part and imaginary part of the first M×(M / 2+1) complex data are retained; the result after the second one-dimensional real number IFFT transform is cached, and the real part and imaginary part of the first M / 2+1 complex data are retained; wherein M is the number of points of the FFT transform and the IFFT transform.
2. The method according to claim 1, It is characterized in that The two-dimensional real number IFFT calculation module is used to process the block frequency domain point multiplication result to obtain time domain feature data, including: When performing multi-channel parallel convolution calculations in the convolution layer, the block frequency domain point multiplication results of all channels are accumulated and then processed using a two-dimensional real number IFFT calculation module to obtain time domain feature data.
3. The method according to claim 1, It is characterized in that The dot product of the block frequency domain feature data and the block frequency domain kernel function is calculated by using a dual-port buffer + single real number multiplier to obtain a block frequency domain dot product result, including: When performing multi-channel parallel convolution calculations within the convolution layer, a shared two-dimensional FFT transform is used to perform point multiplication on the block frequency domain feature data and the kernel functions of the FFT transforms of different channels; the block frequency domain point multiplication result of each channel is obtained.
4. The method according to claim 1, It is characterized in that A quadratic value is used as a point value for FFT transformation and IFFT transformation, and the point value is not greater than 8.
5. The method according to claim 4, It is characterized in that Using short-point real number FFT transform, the rotation factors in the butterfly operation include: 0, ±1 and ; Among them, the multiplication with a rotation factor of 1 is implemented using an adder-subtractor; the rotation factor is The multiplication is implemented by arithmetic left shift followed by accumulation or by constant multiplier IP core.
6. The method according to claim 1, It is characterized in that In the two-dimensional real number FFT transformation calculation module, the two-dimensional FFT transformation is implemented by a real number FFT transformation method of first row and then column, or by a real number FFT transformation method of first column and then row; In the two-dimensional real number IFFT calculation module, the two-dimensional IFFT transformation is implemented by using a real number IFFT transformation method of columns first and rows later, or by using a real number IFFT transformation method of rows first and columns later.
