A three-dimensional convolutional neural network accelerator and method over a complex field

Abstract

The application discloses a three-dimensional convolutional neural network accelerator and method on a complex field, and the accelerator comprises: a cache unit, which is used for storing input features, output features and weight data in the complex field; an AXI DMA unit, which is used for data transmission between the accelerator and an off-chip memory; a calculation unit, which is used for accelerating the calculation of the convolution layer and the full connection layer; a post-processing unit, which is used for calculating the fused quantization layer, the pooling layer, the batch normalization layer and the activation layer; and a control unit, which is used for controlling and scheduling the working states of the cache unit, the AXI DMA unit, the calculation unit and the post-processing unit. The application can significantly improve the performance and energy efficiency of 3D CNN deployment.

Description

Technical Field

[0001] This invention belongs to the field of convolutional neural network technology, and specifically relates to a three-dimensional convolutional neural network accelerator and method. Background Technology

[0002] In recent years, deep convolutional neural networks (CNNs) have achieved great success in image processing. However, when processing higher-dimensional data such as video, traditional two-dimensional CNNs cannot effectively capture the temporal information, thus failing to achieve satisfactory results. The advent of three-dimensional CNNs has solved this problem. Through three-dimensional convolution, they can simultaneously capture spatiotemporal information in video, playing a significant role in video classification and medical image analysis. However, compared to two-dimensional CNNs, three-dimensional CNNs have much larger storage and computational overhead, posing a serious challenge to their deployment in embedded and edge computing scenarios.

[0003] To address this issue, the industry has begun exploring the use of specialized hardware to accelerate 3D CNN algorithms. In the cloud, GPUs have become the mainstream hardware acceleration platform due to their high computational parallelism and high memory bandwidth. At the edge, due to limitations in resources and power consumption, ASIC- and FPGA-based hardware acceleration technologies are commonly used. These technologies improve the deployment efficiency of 3D CNNs by providing higher parallelism at the computational level and maximizing data reuse at the memory access level. ASICs are application-specific integrated circuit chips that offer the highest performance, lowest power consumption, and smallest area compared to other platforms. FPGAs are reconfigurable, have shorter development cycles and are easier to develop than ASICs, while also offering high performance and low power consumption. Therefore, FPGA-based accelerators can achieve higher performance and energy efficiency and are better suited to the rapid iteration of deep learning algorithms.

[0004] Currently, mainstream hardware accelerators primarily employ structures such as vector inner product units (VINs), systolic arrays (SIs), and row buffers. VINs expand the input and output channels of convolution to achieve parallelism in the input and output dimensions; a typical example is DianNao from the Institute of Computing Technology, Chinese Academy of Sciences. SIs include three data streams: fixed input, fixed output, and fixed weights. Data is reused by transferring data between adjacent physical layers (PEs), thereby improving accelerator performance and energy efficiency; a typical example is Google's Tensor Processor. Row buffer architectures cache data... K × K The input features of the window enable parallel computation of the convolution kernel dimension. Under this hardware architecture, feature map pipeline processing can be easily implemented, thus achieving a high throughput.

[0005] Research has shown that performing neural network computations in the complex domain has many advantages. However, there is currently a lack of research on accelerators for three-dimensional convolutional neural networks in the complex domain. Summary of the Invention

[0006] To address the lack of existing research on 3D convolutional neural network accelerators in the complex domain, this invention provides a 3D convolutional neural network accelerator and method in the complex domain, which can significantly improve the performance and energy efficiency of 3D CNN deployment.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] The first aspect of this invention provides a three-dimensional convolutional neural network accelerator, wherein the three-dimensional convolutional neural network includes convolutional layers, fully connected layers, pooling layers, activation layers, and batch normalization layers, characterized in that the accelerator includes:

[0009] A cache unit is used to store input features, output features, and weight data in the complex field;

[0010] The AXI DMA unit is used for data transfer between the accelerator and off-chip memory.

[0011] A computing unit, wherein the computing unit is used to accelerate the computation of the convolutional layer and the fully connected layer;

[0012] The post-processing unit is used to calculate the fused quantization layer, pooling layer, batch normalization layer, and activation layer.

[0013] The control unit is used to control and schedule the working status of the cache unit, AXI DMA unit, computing unit and post-processing unit.

[0014] The cache unit includes:

[0015] An input feature cache unit is used to store input features in the complex field;

[0016] An output feature cache unit is used to store output features in the complex field;

[0017] A weight caching unit is used to cache weights in the complex field.

[0018] The computing unit includes:

[0019] The computational unit matrix includes multiple computational units (PEs), and the multiple computational units (PEs) are arranged in an X-shape. 𝑚 A two-dimensional matrix of size / k×k is arranged, and each operational unit PE includes k𝑛 / 2 parallel complex multipliers and one for 𝑇 𝑛 / k parallel complex multipliers output a complex addition tree for summation, where k 𝑚 This refers to the block size of the output channel. B Let be the block size of the two-dimensional matrix, 𝑇 𝑛 This is the block size of the input channel;

[0020] An address generator is used to generate address data for input features, output features, and weight data.

[0021] The PE controller is used to control the working state of the arithmetic unit (PE) and the address generator.

[0022] The AXI DMA unit includes:

[0023] A data packaging unit is used to package the output data of the cache unit to increase the output data bandwidth;

[0024] A data disassembly unit is used to disassemble the data in the off-chip memory to obtain the data required by the accelerator;

[0025] The AXI DMA controller is used to control the working status of the data packing unit and the data unpacking unit.

[0026] A second aspect of this invention discloses a method for accelerating a three-dimensional convolutional neural network, comprising the following steps:

[0027] Quantization of a 3D convolutional neural network;

[0028] Deploying a 3D convolutional neural network;

[0029] The three-dimensional convolutional neural network is accelerated using a three-dimensional convolutional neural network accelerator as described in the first aspect and any possible design thereof.

[0030] The quantization of the three-dimensional convolutional neural network includes:

[0031] Scaling factors for calculating the real and imaginary parts of the weight values Scaling factors of the real and imaginary parts of the activation value ,

[0032] Based on the scaling factor of the real and imaginary parts of the weight values Scaling factors of the real and imaginary parts of the activation value Calculate pseudo-quantization operators, which include the quantization operator CQuant and the dequantization operator CDequant.

[0033] Insert pseudo-quantization operators into the computation graph of a 3D convolutional neural network;

[0034] in,

[0035] ,

[0036] middle l =1 and 2 represent the scaling factors of the real and imaginary parts of the weight value, respectively.

[0037] a l middle l =1 and 2 represent the real and imaginary parts of the activation value, respectively. β ∈[0,1];

[0038] ,

[0039] ,

[0040] in, z For the complex number to be quantized, z r , z i They represent z The real and imaginary parts, j It is the square root of -1;

[0041] Quant is a symmetric quantization operator over the real number field. x )=clamp([ x / s ],-127,127),clamp( x , a , b ) used to x Value constraints a , b Between, if x Less than a Then return a ,like x Greater than b Then return b Otherwise return x ;

[0042] Dequant is the antisymmetric quantization operator in the real number field. x )= x × s .

[0043] The process of deploying the 3D convolutional neural network also includes:

[0044] Obtain the complex sequence of weights and activations after FFT transformation;

[0045] Based on the complex conjugate symmetry of the real number FFT, the storage and computation of complex number sequences are compressed. When N is even, only the ∠0, ∠1, ..., ∠N in the complex number sequence need to be stored. 𝑁 / 2 There are a total of 𝑁 / 2+1 complex numbers, and we can further combine the real numbers X0 and X... N / 2 Packaged into a plural number 𝑋0+ j 𝑋 𝑁 / 2 In terms of computation, only the product of the first N / 2+1 complex numbers needs to be calculated.

[0046] The process of deploying the 3D convolutional neural network also includes an optimization step for complex number multiplication:

[0047] Get the first complex number to be multiplied. z 1 and second complex number z 2, First complex number z 1= a + bj , second plural z 2= c + DJ , a , b , c , d All are real numbers. j It represents the square root of -1;

[0048] The first complex number z 1 and second complex number z Multiplication of 2 is transformed into calculation ,in, , , .

[0049] The process of deploying the 3D convolutional neural network also includes an optimization step for complex number multiplication:

[0050] Obtain the third complex number 𝑥, the fourth complex number 𝑤1, and the fifth complex number 𝑤2 to be multiplied. The third complex number 𝑥 needs to be multiplied by the fourth complex number 𝑤1 and the fifth complex number 𝑤2 respectively, 𝑥 = 𝑎 + 𝑏 j , 𝑤1=𝑥1+ j 𝑦1, 𝑤2 = 𝑥2 + j 𝑦2, of which a , b , x 1, y 1, x 2,y Both 2 are real numbers. j It represents the square root of -1;

[0051] Transform the product of the third complex number 𝑥 and the fourth complex number 𝑤1 into ,in, , , ,

[0052] Transform the product of the third complex number 𝑥 and the fifth complex number 𝑤2 into ,in, , , .

[0053] Shift the second multiplier r1 left by 18 bits and sign-extend the second multiplier r2 to 27 bits, then sum the two. r 1<<18+ r 2, where r1 and r2 are both real numbers;

[0054] Calculate the product of r1<<18+r2 and the first multiplier r3, and obtain... o = x ×(r1<<18+r2);

[0055] from o The results of separating r3×r1 and r3×r2 are obtained.

[0056] Compared with the prior art, the present invention has at least the following advantages and beneficial effects:

[0057] 1. The accelerator in this solution adopts optimization methods such as cyclic block division, cyclic unrolling, and double buffering operation, which effectively improves the performance and energy efficiency of 3D convolutional neural network deployment.

[0058] 2. This approach takes into account the characteristics of complex number operations and adopts a series of optimization measures to effectively reduce storage space and computational load, reduce computational complexity by using a fast algorithm for complex number multiplication, reduce hardware resource consumption, and improve the accelerator's computational efficiency. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram of the accelerator of the present invention.

[0061] Figure 2 This is a schematic diagram illustrating the optimization of complex conjugate symmetry in this invention;

[0062] Figure 3 This is a schematic diagram of the INT9 multiplication DSP optimization of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0064] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0065] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other.

[0066] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0067] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0068] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0069] Research shows that performing neural network computations in the complex domain has many advantages.

[0070] For example, traditional 2D / 3D FFT algorithms can significantly reduce the computational complexity of 2D / 3D convolution operations. This algorithm first zero-pads the feature map and weights to the same size, then transforms them to the frequency domain using 2D / 3D FFT, performs element-wise multiplication in the frequency domain, and accumulates along the channels. Finally, it transforms the result back to the time domain using 2D / 3D IFFT. Using this algorithm to accelerate convolution operations can reduce the computational complexity by nearly [percentage missing]. K 2 or K 3 The computational cost is several times higher. However, the above method requires a large number of zero-padding operations when the feature map and convolution kernel sizes differ significantly, thus increasing the storage overhead of the computation process. To address this issue, researchers proposed Overlap-and-Add (OaA). OaA first divides the feature map into blocks, then zero-padding each block and the corresponding convolution kernel to the same size. After transforming both to the frequency domain using FFT, element-wise multiplication and accumulation operations are performed in the frequency domain, and finally, the model is transformed back to the time domain. Because OaA divides the feature map into blocks, the size difference between the feature map and the convolution kernel is no longer significant, and the additional storage overhead during computation is alleviated.

[0071] Another approach to neural networks in the complex domain is based on cyclic matrix compression. After compressing the neural network using a cyclic matrix, the computation can be further transformed to the frequency domain using 1D FFT, according to the circular convolution theorem, thus achieving acceleration. Unlike traditional 2D / 3D FFT algorithms, 1D FFT based on cyclic matrix compression does not involve zero-padding during computation, significantly reducing both storage and computational overhead. Furthermore, this method can balance compression ratio and accuracy by adjusting the size of the cyclic matrix; a larger cyclic matrix provides a higher compression ratio but also results in greater accuracy loss, and vice versa.

[0072] Besides complex-domain neural networks that are transformed to the frequency domain via FFT, another type of complex-domain neural network is built directly on the complex domain. Related research shows that complex numbers are easier to optimize during neural network training, have better generalization characteristics, faster learning capabilities, and allow for noisy, robust memory mechanisms. For example, in recurrent neural networks, using complex numbers can give the network richer representational capabilities. Therefore, complex-domain neural networks have begun to attract increasing attention. In this type of network, the input feature map, weights, and output feature map are all composed of complex numbers. Corresponding to neural networks in the time domain, this type of network also has complex convolutional layers, complex fully connected layers, complex pooling layers, complex activation function layers (activation layers), and complex batch normalization layers (input layers).

[0073] Because FPGA on-chip storage resources are limited and cannot accommodate feature maps and weights all at once, this invention employs cyclic block optimization in the accelerator design. For ease of explanation, let's use the following convention: 𝑛 , 𝑚 These represent the block sizes for the accelerator input and output channels, respectively. d , 𝑟 , 𝑐 To determine the block size in the time, height, and width dimensions of the output feature map, B This refers to the block size of the cyclic matrix. Based on this, a first aspect of the present invention provides a three-dimensional convolutional neural network accelerator, wherein the three-dimensional convolutional neural network includes convolutional layers, fully connected layers, pooling layers, activation layers, and batch normalization layers. This accelerator, as... Figure 1 As shown, the accelerator includes a cache unit, an AXI DMA unit, a computation unit, a post-processing unit, and a control unit. The cache unit stores input features, output features, and weight data in the complex domain. The AXI DMA unit is used for data transfer between the accelerator and off-chip memory. The computation units (PEs) accelerate the computation of the convolutional and fully connected layers. The post-processing unit computes the fused quantization, pooling, batch normalization, and activation layers, mainly including linear transformation, truncation, and rounding operations. The control unit controls and schedules the working states of the cache unit, AXI DMA unit, computation units (PEs), and post-processing unit.

[0074] To facilitate providing matching on-chip memory access bandwidth to the computing unit, the cache unit includes an input feature cache unit, an output feature cache unit, and a weight cache unit. The input feature cache unit stores input features in the complex domain; the output feature cache unit stores output features in the complex domain; and the weight cache unit caches weights in the complex domain. The input feature cache, output feature cache, and weight cache are all composed of multiple banks, with data from different channels stored in different banks, thus enabling parallel reading and writing of data.

[0075] To mask data transmission time, the accelerator can employ a double-buffering setup, with two buffers each for the input feature buffer, output feature buffer, and weight buffer. Two buffers are set up for each type of data. When the producer writes data to buffer 1 or 2, the consumer reads data from buffer 2 or 1 for subsequent use, thus achieving coarse-grained pipelining and further increasing the accelerator's throughput. The accelerator's operation consists of three stages: data loading, data computation, and data write-back. Two buffers are set up for the input feature map, weights, and output feature map, and ping-pong operations are used to mask data transmission time, thereby achieving coarse-grained pipelining between the three stages.

[0076] For example, the computing units (PEs) include a computation unit matrix, an address generator, and a PE controller. The computation unit matrix includes multiple computation units (PEs), and the multiple computation units (PEs) are arranged in an array. 𝑚 A two-dimensional matrix of size / k×k is arranged, and each operational unit PE includes k 𝑛 / 2 parallel complex multipliers and one for 𝑇 𝑛 / k parallel complex multipliers output complex addition trees for summation, such a structure has a total of k 𝑚 / 𝐵 × 𝐵 = 𝑇 𝑚 Therefore, the total parallelism is . 𝑚 ×𝑇 𝑛 / 𝐵, meaning that each cycle can complete 𝑇 𝑚 ×𝑇 𝑛 / k complex multiplication operations. The address generator is used to generate address data for input features, output features, and weight data; the PE controller is used to control the working state of the computing unit PE and the address generator.

[0077] This accelerator employs a cyclic block optimization method, dividing the output feature map into blocks based on five dimensions: time, height, width, and the input and output channels of the weights. The block sizes are as follows: T d , T r , Tc , T n , T m Meanwhile, the accelerator has been optimized in terms of size in both the input and output channels. T n , T m The cyclical expansion.

[0078] The AXI DMA unit is used to move data between off-chip memory and the accelerator. It includes a data packing unit, a data decomposition unit, and an AXI DMA controller. The data packing unit is used to pack the output data of the cache unit to increase the output data bandwidth. The data decomposition unit is used to decompose the data in the off-chip memory to obtain the data required by the accelerator. The AXI DMA controller is used to control the working state of the data packing unit and the data decomposition unit.

[0079] Based on the above accelerator structure, the second aspect of the present invention discloses a three-dimensional convolutional neural network acceleration method, including steps S01 to S04.

[0080] Step S01: Quantize the three-dimensional convolutional neural network.

[0081] A three-dimensional convolutional neural network over the complex domain was trained using INT8 quantization, further reducing storage and computational overhead. The INT8 quantization training over the complex domain involves performing symmetric quantization on the real and imaginary parts of the complex number based on the maximum absolute value, with these two operations having independent scaling factors. Therefore, it can maintain model accuracy while reducing computational and storage overhead. The quantization step reduces the model's storage and computational costs. Specifically, this quantization step includes steps S011 to S013.

[0082] Step S011: Calculate the scaling factors for the real and imaginary parts of the weight values. Scaling factors of the real and imaginary parts of the activation value .

[0083] in,

[0084] ,

[0085] ,

[0086] middle l =1 and 2 represent the scaling factors of the real and imaginary parts of the weight value, respectively.

[0087] a l middlel =1 and 2 represent the real and imaginary parts of the activation value, respectively. β ∈[0,1]. β It plays a moderating role; the smaller the value, the greater the influence of historical values, and vice versa, the current value dominates.

[0088] The scaling factor for the real and imaginary parts in this step Both are based on weights The statistical information is dynamically calculated during training. To smooth the activation scaling factor and prevent drastic fluctuations during training, this paper employs an exponential moving average method to update it. .

[0089] Furthermore, since the rounding function is not differentiable during backpropagation, a straight-through estimator (STE) can be used to estimate its derivative, as shown in Equation (1).

[0090] (1)

[0091] Step S012: Based on the scaling factors of the real and imaginary parts of the weight values. Scaling factors of the real and imaginary parts of the activation value Calculate pseudo-quantization operators, which include the quantization operator CQuant and the dequantization operator CDequant.

[0092] in, ,

[0093] ,

[0094] z For the complex number to be quantized, z r , z i They represent z The real and imaginary parts, j It is the square root of -1;

[0095] Quant is a symmetric quantization operator over the real number field. x )=clamp([ x / s ],-127,127),clamp( x , a , b ) used to x Value constraints a , b Between, ifx Less than a Then return a ,like x Greater than b Then return b Otherwise return x ;

[0096] Dequant is the antisymmetric quantization operator in the real number field. x )= x × s .

[0097] In order to reduce the impact of quantization on model accuracy during the quantization process, this scheme uses independent scaling factors for weights and activations, as well as for real and imaginary parts.

[0098] Step S013: Insert the pseudo-quantization operator into the computation graph of the three-dimensional convolutional neural network to achieve quantization-aware training.

[0099] Step S02: Optimize the quantized 3D convolutional neural network. The optimization of the 3D convolutional neural network is based on the characteristics of complex number operations. Specifically, this includes optimization using the conjugate symmetry of the real number sequence after FFT, optimization of a fast algorithm based on complex multiplication, and optimization of low-bit-width complex multiplication based on DSP.

[0100] Step S021: This step mainly utilizes the conjugate symmetry of the real number sequence after FFT for optimization. Complex domain 3D convolutional neural networks include traditional ones based on 3D FFT acceleration algorithms, those based on cyclic matrix compression and 1D FFT acceleration, and pure complex 3D convolutional neural networks. In the above complex domain neural networks, the first two are obtained from the real number domain through FFT transformation; therefore, their activations and weights in the complex domain satisfy conjugate symmetry. This invention utilizes this property to further optimize the storage and computation of the accelerator.

[0101] Without loss of generality, taking 1D FFT as an example, a sequence of real numbers of length 𝑁 x =[ x 0, x 1, …, x N-1 ] T After FFT transformation, the resulting complex sequence is 𝑋=[ X 0, X 1, …, X N-1 ] T It satisfies conjugate symmetry.

[0102]

[0103] ,

[0104] Where * denotes the conjugate of the complex number, i.e., (x + y). j ) ∗ =𝑎−𝑏 j Therefore, when storing the weights (which can be pre-converted to the frequency domain) and activations in the frequency domain, only a subset of their values ​​needs to be stored. For even numbers, only α0, α1, ..., α need to be stored. 𝑁 / 2 There are a total of 𝑁 / 2+1 complex numbers, taking into account 𝑋0 and 𝑋 N / 2 Since both are real numbers, we can consider 0 and 1 as real numbers. 𝑁 / 2 Further packaged into a plural number 𝑋0+ j 𝑋 𝑁 / 2 This would halve the storage space, such as Figure 2 As shown. Furthermore, since the product of conjugate complex numbers is equal to the conjugate of the product of complex numbers, ,in, z 1 and z 2 is any complex number. * This represents the conjugate of a complex number. Therefore, when performing element-wise multiplication of complex vectors in the frequency domain, only the product of the first *n / 2+1* terms needs to be calculated, and the product of the last *n / 2-1* terms can be obtained from the complex conjugate of the former, thus reducing the computational load by nearly 50%. The main method in this step is as follows: obtain the complex sequence obtained by FFT transformation of the weights and activations; compress storage and computation based on the conjugate symmetry after real-number FFT. When N is even, only *n0*, *n1*, ..., *n* in the complex sequence need to be stored. 𝑁 / 2 There are a total of 𝑁 / 2+1 complex numbers, and the real numbers X0 and XN / 2 can be further packaged into a complex number 𝑋0+. j 𝑋 𝑁 / 2 In terms of computation, only the product of the first N / 2+1 terms needs to be calculated.

[0105] Step S022: This step mainly focuses on optimizing a fast algorithm for complex number multiplication. Complex number multiplication is the core operation of complex-domain 3D convolutional neural networks. However, unlike real number multiplication, complex number multiplication has a more complex form. Let complex numbers... z 1= a + bj , z 2= c + DJ , a , b , c , d All are real numbers. j To represent the square root of -1, then z 1 and z The product of 2 is as follows:

[0106]

[0107] The above equation shows that a complex number multiplication of one degree consists of four real number multiplications and three real number additions.

[0108] To reduce the computational complexity of complex number multiplication, the specific steps of this solution are as follows: Obtain the first complex number to be multiplied. z 1 and second complex number z 2, First complex number z 1= a + bj , second plural z 2= c + DJ , a , b , c , d All are real numbers. j Represent the square root of -1; [The first complex number] z 1 and second complex number z Multiplication of 2 is transformed into calculation ,in, , , .

[0109] After the above optimization, the computational complexity of complex number multiplication, which originally involved 4 real number multiplications and 3 real number additions, can be reduced to 3 real number multiplications and 5 real number additions. Since the computational complexity of addition is much lower than that of multiplication, this optimization can effectively reduce the computational complexity of complex number multiplication, thereby reducing hardware resource consumption (such as DSP slices) by about 25%.

[0110] Step S023 primarily focuses on optimizing low-bit-width complex multiplication based on the DSP. The DSP is a crucial computing resource in FPGAs. Taking Xilinx FPGAs as an example, in the UltraScale / UltraScale+ series FPGAs, the DSP48E2 Slice includes a 27×18-bit multiplier and a 48-bit adder. Due to its support for a large bit width, it can be further optimized to improve DSP utilization when the data bit width involved in the computation is small and data sharing exists. Meanwhile, 3D-CNNs exhibit rich data reuse, including input feature map reuse, filter reuse, and convolutional reuse, making data sharing possible. Furthermore, DNN model quantization has become an essential technique for high-performance DNN inference, and related research has demonstrated that low-bit-width quantization such as INT8 and even INT4 can still maintain good model accuracy on many neural networks.

[0111] Based on the above three observations, this invention takes INT8 quantization as an example to illustrate an optimization method based on DSP packing technology in the complex domain.

[0112] Now consider three complex numbers: 𝑥 = 𝑎 + 𝑏 j , 𝑤1=𝑥1+ j 𝑦1, 𝑤2 = 𝑥2 + j 𝑦2, of which a , b , x 1, y 1, x 2, y Both 2 are real numbers. j To represent the square root of -1, we need to calculate 𝑜1=𝑥×𝑤1 and 𝑜2=𝑥×𝑤2, meaning that one multiplier is shared, which can be obtained by reusing the input feature map in the convolution calculation.

[0113] To reduce DSP resource usage, this step includes:

[0114] Obtain the third complex number 𝑥, the fourth complex number 𝑤1, and the fifth complex number 𝑤2 to be multiplied. The third complex number 𝑥 needs to be multiplied by the fourth complex number 𝑤1 and the fifth complex number 𝑤2 respectively, 𝑥 = 𝑎 + 𝑏 j , 𝑤1=𝑥1+ j 𝑦1, 𝑤2 = 𝑥2 + j 𝑦2, of which a , b , x 1,y 1, x 2, y Both 2 are real numbers. j It represents the square root of -1;

[0115] Transform the product of the third complex number 𝑥 and the fourth complex number 𝑤1 into ,in, , , ,

[0116] Transform the product of the third complex number 𝑥 and the fifth complex number 𝑤2 into ,in, , , .

[0117] 𝐴1 and 𝐴2 share the multiplier 𝑎+𝑏, 𝐵1 and 𝐵2 share the multiplier 𝑏, and 𝐶1 and 𝐶2 share the multiplier 𝑏−𝑎. However, when both the real and imaginary parts are 8-bit signed integers, 𝑎+𝑏, 𝑏−𝑎, 𝑥1+𝑦1, and 𝑥2+𝑦2 all need to be represented by 9-bit signed integers. Therefore, the problem can be transformed into: how to optimize two signed INT9 multiplications with one shared multiplier to reduce the usage of DSP resources.

[0118] The problems of A1 and A2, B1 and B2, and C1 and C2 are abstracted into multiplying one number with two other numbers. Further optimization is needed to reduce DSP resource consumption. r1, r2, and r3 are all of type INT9. To calculate r3×r1 and r3×r2, how can this be optimized to reduce DSP resource consumption? This step involves packaging r1 and r2 into a larger bit-width number, multiplying it with r3, and then separating r3×r1 and r3×r2 from the result. The key here is that r3×r1 and r3×r2 can be separated. Since r3×r1 or r3×r2 requires at most 18 bits to store, and the core multiplier of the DSP48E2 Slice in the Xilinx ZCU102 FPGA has a bit width of 18×27, r1 is first left-shifted by 18 bits, and r2 is sign-extended to 27 bits. The two are then summed to obtain... r 1<<18+ r 2, then multiply by r3, that is o = r3×( r 1<<18+ r 2), finally from the product o The result of splitting r3×r1 and r3×r2: r3× r 1= o 35:18 +o 17 , r3×r 2= o 17:0 , o 35:17 Indicates will o The 35th to 18th bits are extracted and treated as an 18-bit signed integer. o 17:0 Similarly, this means that... o The 17th to 0th bits are extracted and treated as an 18-bit signed integer, such as... Figure 3 As shown. Through the above optimization, it is possible to achieve the goal of using one DSP48E2 Slice to calculate two INT9 multiplications (one of which is shared). This fully utilizes the computing power of the DSP48E2 Slice, significantly reduces the usage of DSP resources, reduces the resource consumption of the accelerator, and improves the overall computing efficiency.

[0119] Step S03: Deploy the 3D convolutional neural network. The deployment steps can be performed using existing methods and will not be elaborated upon in this solution.

[0120] Step S04: Accelerate the three-dimensional convolutional neural network using a three-dimensional convolutional neural network accelerator as described in the first aspect and any possible design thereof.

[0121] This invention employs an accelerator and acceleration method to accelerate 3D convolutional neural networks in the complex domain, effectively alleviating the problem of excessive storage and computational overhead during the deployment of 3D convolutional neural networks. Quantization-based perceptual training of the 3D CNN was performed in the complex domain, and an efficient hardware architecture was designed. Considering the characteristics of complex number operations, a series of optimization measures were adopted to further reduce storage and computational overhead and improve accelerator performance, including: utilizing the conjugate symmetry after FFT of real number sequences to reduce storage and computational overhead; utilizing a fast algorithm for complex multiplication to further reduce computational load; and utilizing DSP packaging technology in FPGAs, combined with the rich data reuse in 3D CNNs, to reduce the consumption of DSP and other resources and improve the accelerator's computational efficiency.

[0122] To evaluate the technical advantages of this invention, we selected two 3D CNN models, C3D and 3D ResNet-18, for experiments. During training, INT8 quantization was achieved by inserting pseudo-quantization nodes. In the experiments, the size of the cyclic matrix was set to 8, with corresponding accuracy losses of 1.25% and 1.71%, respectively, which are within an acceptable range. The hardware platform used in the experiments was a Xilinx ZCU102 FPGA, and the development tool was Vivado HLS 2019.2. We designed corresponding hardware accelerators for C3D and 3D ResNet-18. For C3D, the accelerator configuration is as follows ( Tn , T m ) = (16, 64), ( T d , T r , T c = (2, 7, 7). For 3D ResNet-18, we designed two hardware acceleration cores to accelerate 1×1×1 and 3×3×3 convolutions, with parallelism of (k) respectively. 𝑛 , 𝑇 𝑚 Given (8, 32) and (32, 32), output the block size of the feature map (k). d , 𝑇 𝑟 , 𝑇 𝑐 The values ​​are all (2, 7, 7). External memory transmits data to the accelerator via the AXI bus. The AXI bus interface has a bit width of 128 bits, allowing for the parallel reading of 8 complex numbers at a time (each complex number has 8 bits for both the real and imaginary parts, totaling 16 bits). After the accelerator design was completed, Xilinx Vivado 2019.2 was used for synthesis, placement and routing, and bitstream generation. The final clock frequency for both accelerators was 200MHz. Table 1 shows the final resource consumption and power consumption of the two accelerators, and Table 2 provides a comparison of the two accelerators with other related work.

[0123] Table 1. Accelerator resource and power consumption.

[0124]

[0125] Table 2 Comparison with other works

[0126]

[0127] As shown in Table 2, the power consumption of the C3D and 3D ResNet-18 accelerators is 5.461W and 6.477W, respectively, lower than all other works listed in the table. This is mainly because the efficient compression scheme of this invention greatly reduces the power consumption generated by memory access and computation. Furthermore, the two accelerators consume only 452 and 611 DSP slices, far fewer than other works, yet their throughput reaches 1476.81 GOP / s and 876.27 GOP / s, respectively. Among these works, the C3D accelerator's computational performance of 1476.81 GOP / s is the best, achieving a speedup of 1.1 to 9.15 times. The final performance of the 3D ResNet-18 accelerator was lower than that of the C3D accelerator, reaching 876.27 GOP / s. This performance was higher than that of work [2] and work [3], but lower than that of work [1]. This was mainly because the 3D ResNet-18 network contained 1×1×1 convolutions and convolutions with a stride of 2, resulting in a lower computation-to-memory ratio compared to C3D and a greater difference between layers. Therefore, there was a significant gap between the actual performance and the theoretical value. In terms of computational efficiency, the C3D accelerator achieved 3.267 GOP / s / DSPs, which was 2.47 times that of work [1] and 11.19 to 15.78 times that of the other works in the table. Although the performance of the 3DResNet-18 accelerator was not optimal, it still achieved the highest computational efficiency (1.08 to 6.93 times) compared to the other works in the table, which fully demonstrated the advantages of the complex domain three-dimensional convolutional neural network accelerator designed in this invention in terms of performance and computational efficiency.

[0128] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A three-dimensional convolutional neural network accelerator over the complex domain, the three-dimensional convolutional neural network comprising convolutional layers, fully connected layers, pooling layers, activation layers, and batch normalization layers, characterized in that, The following steps are included before deploying a 3D convolutional neural network: Quantization of a 3D convolutional neural network; Obtain the complex sequence of weights and activations after FFT transformation; When N is even, only the complex sequence 𝑋0, 𝑋1, ..., 𝑋 is stored. 𝑁 / 2 There are a total of 𝑁 / 2+1 complex numbers, and the real numbers X0 and X... N / 2 Packed into a complex number 𝑋0+j𝑋 𝑁 / 2 ; Optimization steps for complex number multiplication: Obtain the first complex number z1 and the second complex number z2 to be multiplied. The first complex number z1 = a + bj and the second complex number z2 = c + dj, where a, b, c, and d are all real numbers and j represents the square root of -1. The multiplication of the first complex number z1 and the second complex number z2 is transformed into a calculation ,in, , , ;and, Obtain the third complex number 𝑥, the fourth complex number 𝑤1, and the fifth complex number 𝑤2 to be multiplied. The third complex number 𝑥 needs to be multiplied by the fourth complex number 𝑤1 and the fifth complex number 𝑤2 respectively, 𝑥=𝑎+𝑏j, 𝑤1=𝑥1+j𝑦1, 𝑤2=𝑥2+j𝑦2, where a, b, x1, y1, x2, y2 are all real numbers, and j represents the square root of -1; Transform the product of the third complex number 𝑥 and the fourth complex number 𝑤1 into ,in, , , , Transform the product of the third complex number 𝑥 and the fifth complex number 𝑤2 into ,in, , , ; The accelerator includes: A cache unit is used to store input features, output features, and weight data in the complex field; The AXI DMA unit is used for data transfer between the accelerator and off-chip memory. A computation unit, used to accelerate the computation of the convolutional layer and the fully connected layer, comprises: a computation unit matrix, wherein the computation unit matrix includes multiple computation units (PEs) arranged in an array. 𝑚 A two-dimensional matrix of size / k×k is arranged, and each operational unit PE includes k 𝑛 / 2 parallel complex multipliers and one for 𝑇 𝑛 / k parallel complex multipliers output a complex addition tree for summation, where k 𝑚 Where B is the block size of the output channel, and B is the block size of the two-dimensional matrix. 𝑛 The input channel is divided into blocks; an address generator is used to generate address data for input features, output features, and weight data; and a PE controller is used to control the working state of the computing unit PE and the address generator. The post-processing unit is used to calculate the fused quantization layer, pooling layer, batch normalization layer, and activation layer. The control unit is used to control and schedule the working status of the cache unit, AXI DMA unit, computing unit and post-processing unit.

2. The three-dimensional convolutional neural network accelerator over the complex domain according to claim 1, characterized in that, The cache unit includes: An input feature cache unit is used to store input features in the complex field; An output feature cache unit is used to store output features in the complex field; A weight caching unit is used to cache weights in the complex field.

3. The three-dimensional convolutional neural network accelerator over the complex domain according to claim 1, characterized in that, The AXI DMA unit includes: A data packaging unit is used to package the output data of the cache unit to increase the output data bandwidth; A data disassembly unit is used to disassemble the data in the off-chip memory to obtain the data required by the accelerator; The AXI DMA controller is used to control the working status of the data packing unit and the data unpacking unit.

4. A method for accelerating three-dimensional convolutional neural networks over the complex domain, characterized in that, Includes the following steps: Quantization of a 3D convolutional neural network; Deploying a 3D convolutional neural network; The three-dimensional convolutional neural network is accelerated using a three-dimensional convolutional neural network accelerator over the complex domain as described in any one of claims 1 to 3.

5. The method for accelerating a three-dimensional convolutional neural network in the complex domain according to claim 4, characterized in that, The quantization of the three-dimensional convolutional neural network includes: Scaling factors for calculating the real and imaginary parts of the weight values Scaling factors of the real and imaginary parts of the activation value , Based on the scaling factor of the real and imaginary parts of the weight values Scaling factors of the real and imaginary parts of the activation value Calculate pseudo-quantization operators, which include the quantization operator CQuant and the dequantization operator CDequant. Insert pseudo-quantization operators into the computation graph of a 3D convolutional neural network; in, , ， When l=1 and 2, they represent the scaling factors of the real and imaginary parts of the weight value, respectively. a l When l = 1 and 2, they represent the real and imaginary parts of the activation value, respectively, and β∈[0,1]. ， ， Where z is the complex number to be quantized, z r z i Let z represent the real and imaginary parts of z, respectively, and j be the square root of -1; Quant is a symmetric quantization operator over the real number field. Quant(x) = clamp([x / s], -127, 127). clamp(x, a, b) is used to constrain the value of x between a and b. If x is less than a, it returns a; if x is greater than b, it returns b; otherwise, it returns x. Dequant is an antisymmetric quantization operator over the real number field, Dequant(x) = x × s.

6. The method for accelerating a three-dimensional convolutional neural network in the complex field according to claim 4, characterized in that: Also includes: Shift the first multiplier r1 left by 18 bits and sign-extend the second multiplier r2 to 27 bits, then sum the two to get r1<<18+r2, where r1 and r2 are both real numbers; Calculate the product of r1<<18+r2 and the third multiplier r3, and get o = r3×(r1<<18+r2); The result of separating r3×r1 and r3×r2 from o.

Images