Three-dimensional neural network processing method, image processing method, system, and storage medium

Abstract

The application discloses a three-dimensional neural network processing method, an image processing method, an image processing system and a storage medium, and belongs to the technical fields of neural networks and image processing. The three-dimensional neural network processing method based on a block circulant matrix model overcomes the irregular memory and calculation problems existing in the previous pruning and other model compression methods by constructing a block circulant matrix model, calculating an acceleration model and a full frequency domain model, and proposes to compress a three-dimensional neural network 3D CNN by using the block circulant matrix, and further utilizes fast Fourier transform FFT to accelerate calculation, so that the storage and calculation compression effects are significantly achieved on the premise of keeping the model structure regular. On this basis, the activation, batch normalization and pooling operations in the frequency domain are introduced, the frequent time domain / frequency domain switching overhead caused by the fast Fourier transform FFT is further eliminated, so that full frequency domain calculation is realized, and the calculation overhead during 3D CNN model reasoning is further reduced.

Description

Technical Field

[0001] This invention relates to three-dimensional neural network processing methods, image processing methods, systems, and storage media, belonging to the field of neural network and image processing technology. Background Technology

[0002] With the advent of the big data era and the improvement of computing power, two-dimensional neural networks (2D CNNs) have achieved state-of-the-art accuracy in image classification. However, when processing high-dimensional data such as videos, image-based 2D CNNs cannot model the temporal information and motion patterns, thus failing to achieve satisfactory results. To achieve accurate classification of high-dimensional data such as videos, researchers proposed using three-dimensional convolution to capture the spatiotemporal information in such data. In 2010, Ji et al. first proposed a three-dimensional convolutional neural network, which performs three-dimensional convolution on adjacent image frames to extract features in the temporal and spatial dimensions. Subsequently, Tran et al. proposed a modern deep architecture, C3D, which has deeper layers than previous 3D neural networks (3D CNNs), thus enabling it to learn on large-scale datasets and achieve optimal results. Since then, 3D CNNs have achieved great success in video analysis, 3D geometric data analysis, and 3D medical image diagnosis. However, corresponding to the improved algorithm performance is the significantly increased model size and computational cost of 3D CNNs. For example, with the same network architecture, 3D ResNet34 has 63.5M parameters and 36.7 GFLOPs of computation, which are 2.95 times and 10.49 times that of 2D ResNet34 (21.5M parameters and 3.5 GFLOPs of computation). Compared to 2D CNNs, 3D CNNs add a time dimension to capture temporal features, resulting in a significantly larger computational overhead. Therefore, how to deploy 3D CNNs with high performance and low power consumption is currently a research hotspot in academia and industry.

[0003] To address this issue, researchers have begun to optimize the storage and computational overhead of 3D CNNs at the algorithmic level using model compression methods. Commonly used model compression methods include pruning, quantization, low-rank decomposition, and fast algorithms. Pruning typically uses a metric to assess parameter importance, removing unimportant parameters to reduce model redundancy, often resulting in a high compression ratio. However, such methods usually involve fine-tuning and retraining, significantly increasing the training burden of 3D convolutional neural networks. Furthermore, pruning can cause irregular distributions of non-zero elements, leading to irregular memory access and unbalanced computational load, making it unfriendly to hardware implementation. Parameter quantization can effectively reduce computational resource consumption, on-chip storage overhead, and off-chip memory access pressure, but the quantization process often causes significant accuracy loss, and its compression effect on computational intensity is not significant.

[0004] Therefore, quantization is generally used as an auxiliary compression method. Low-rank decomposition uses multiple low-rank matrices to approximate high-dimensional tensors. Common decomposition methods include CP decomposition, SVD decomposition, Tucker decomposition, and Tensor-Train decomposition. Low-rank decomposition can significantly reduce the computational complexity of model inference, but it involves computationally expensive decomposition operations and requires a lot of retraining to achieve convergence, which is unacceptable for the already time-consuming training of 3D CNNs.

[0005] Fast algorithms that can be used to accelerate convolutions include Fast Fourier Transform (FFT) and Winograd. These algorithms first transform the feature map and weights to another domain using some transformation (FFT or Winograd transform), then perform calculations in that other domain, and finally transform the result back to the original domain. However, while these methods can effectively reduce the computational cost of the model, they cannot reduce the number of parameters. Furthermore, the Winograd and FFT algorithms are significantly affected by the convolution size and stride; for example, neither can accelerate 1×1×1 convolutions, which account for a major portion of the computation in modern networks such as 3D ResNet and 3D MobileNet.

[0006] In conclusion, the above methods still do not adequately address the processing efficiency issue of 3D CNNs, leaving their deployment in energy- and resource-constrained scenarios with significant performance and energy efficiency challenges. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to provide a three-dimensional neural network processing method that achieves significant storage and computational compression effects by constructing a block-based cyclic matrix model, a computation acceleration model, and a full-frequency domain model, and by using Fast Fourier Transform (FFT) to accelerate computation. Furthermore, it introduces activation, batch normalization, and pooling operations in the frequency domain to achieve full-frequency domain computation, thereby reducing the computational overhead during 3D CNN model inference.

[0008] The second objective of this invention is to provide an image processing method that uses a block-based cyclic matrix-based three-dimensional neural network processing method to process video images. This method can deploy 3D CNN with high performance and low power consumption, thereby achieving accurate classification and recognition of high-dimensional data such as videos. The solution is scientific, reasonable, practical, and easy to implement.

[0009] The third objective of this invention is to provide a three-dimensional neural network processing system that achieves significant storage and computation compression effects by setting up a block cyclic matrix module, a computation acceleration module, and a full-frequency domain module, and using Fast Fourier Transform (FFT) to accelerate computation while maintaining the regularity of the module structure. Furthermore, it introduces activation, batch normalization, and pooling operations in the frequency domain to realize full-frequency domain computation and reduce computational overhead.

[0010] The fourth objective of this invention is to provide a method applicable to various types of convolutional kernel sizes and strides, with wide applicability and strong versatility, capable of effectively solving the processing efficiency problem of 3D CNNs, enabling its deployment in energy- and resource-constrained scenarios for 3D neural network processing methods, image processing methods, systems, and storage media.

[0011] To achieve one of the above objectives, the first technical solution of the present invention is as follows:

[0012] A three-dimensional neural network processing method based on block cyclic matrices includes the following:

[0013] By using a pre-built block cyclic matrix model, the weights of the neural network are represented by the block cyclic matrix, and the fully connected layers and / or convolutional layers of the neural network are compressed to obtain compressed fully connected layers and / or convolutional layers.

[0014] By using a pre-built computation acceleration model and employing Fourier transform, the compressed fully connected layer and / or convolutional layer are computed at an accelerated speed to obtain frequency domain computation results.

[0015] Based on the pre-constructed full-frequency domain model, frequency domain operators are selected, and frequency domain calculation results are subjected to frequency domain activation, batch normalization, and pooling operations to complete the three-dimensional neural network processing based on the block cyclic matrix.

[0016] Through continuous exploration and experimentation, this invention overcomes the problems of memory access and computational irregularities inherent in previous model compression methods such as pruning by constructing a block-based cyclic matrix model, a computational acceleration model, and a full-frequency domain model. It proposes using a block-based cyclic matrix to compress 3D neural networks (CNNs) and further utilizes Fast Fourier Transform (FFT) to accelerate computation, achieving significant storage and computational compression effects. Building upon this, activation, batch normalization, and pooling operations in the frequency domain are introduced to further eliminate the frequent time-domain / frequency-domain switching overhead caused by FFT, thereby achieving full-frequency domain computation and further reducing the computational overhead during 3D CNN model inference.

[0017] Furthermore, this invention employs a 3D neural network processing method based on block-based cyclic matrices, which is applicable to various types of convolutional kernel sizes and strides, has a wide range of applications, and possesses strong versatility. In addition, this 3D neural network processing method does not introduce additional computational complexity into the model training process. The solution is scientifically sound, practical, and easy to implement, effectively addressing the processing efficiency issue of 3D CNNs and enabling its deployment in energy- and resource-constrained scenarios.

[0018] As a preferred technical measure:

[0019] The method for constructing a block-based cyclic matrix model is as follows:

[0020] Step 1, settings × matrix , is used to represent a circular matrix;

[0021] The second step is to construct the elements of matrix A if and only if each row of elements is the result of the previous row of elements shifted one position to the right. The entire matrix A is represented by storing the elements of the first column of matrix A, and the vector formed by the elements of the first column is the generating vector of the cyclic matrix.

[0022] As a preferred technical measure:

[0023] The method for constructing the computational acceleration model is as follows:

[0024] Based on the circular convolution algorithm, the multiplication of circular matrices and arbitrary vectors is accelerated;

[0025] The circular matrix is × matrix ;

[0026] Vector is A vector of ×1 ;

[0027] The calculation formula for the circular convolution algorithm is as follows:

[0028]

[0029] in, It is a matrix The generated vectors, FFT is the Fast Fourier Transform, IFFT is the Inverse Fast Fourier Transform, and ⊙ represents element-wise multiplication.

[0030] As a preferred technical measure:

[0031] The method for constructing the full-frequency domain model is as follows:

[0032] Based on the computational complexity and accuracy loss of frequency domain operators, CReLU, CBN, and CMaxPool are adopted as frequency domain operators, and their expressions are shown below:

[0033]

[0034]

[0035]

[0036] ReLU, BN, and MaxPool are the activation, batch normalization, and max pooling operations on the real number field, respectively. a b is the value in the real number field, and b is the value in the imaginary number field.

[0037] As a preferred technical measure:

[0038] The method for compressing the fully connected layers of a neural network is as follows:

[0039] Step 11, obtain the weight matrix of the fully connected layer. W Weight matrix W The size is M × N ;

[0040] Step 12, with dimensions of M × N weight matrix W Divided into p × q indivual k × k square array W ij ;

[0041] = / , = / If it cannot be divided evenly, then... W Continue filling until the result is divisible;

[0042] Step 13, arrange the square array W ij It is constructed as a cyclic matrix to compress the number of parameters in a neural network.

[0043] As a preferred technical measure:

[0044] The method for accelerating computation of the compressed fully connected layer is as follows:

[0045] S11, based on the cyclic matrix, calculate the fully connected layer. a = W x The calculation is performed using block matrix-vector multiplication, and the formula is as follows:

[0046] ,

[0047] in, a To output feature vectors ，W It is a block-based cyclic matrix. The input feature vector; This is a subvector of the output feature vector. W ij It is a circular matrix. A subvector of the input feature vector;

[0048] S12 accelerates the block matrix-vector multiplication using Fast Fourier Transform (FFT) to reduce the computational cost of the fully connected layer. The calculation formula is as follows:

[0049]

[0050] S13, utilizing the linearity of Fast Fourier Transform (FFT) / Inverse Fast Fourier Transform (IFFT), moves the Inverse Fast Fourier Transform (IFFT) outside the summation symbol, reducing the number of IFFT calls from... q The formula for reducing it to 1 is as follows:

[0051]

[0052] Here, FFT stands for Fast Fourier Transform, and IFFT stands for Inverse Fast Fourier Transform.

[0053] As a preferred technical measure:

[0054] It also includes accelerating the backpropagation process of the neural network using a block-based cyclic matrix, the expression of which is as follows:

[0055]

[0056]

[0057] in, L For loss function, Subvectors of the output feature vector a i The l One element, Subvectors of the input feature vector x j The l One element, The generating vector corresponding to the circular matrix is w ij ;

[0058] and It has a structure with block cyclic matrices, and its generating vectors are respectively =( , , ..., )and w ij ;

[0059] Will and The process is accelerated by sequentially performing Fast Fourier Transform (FFT), multiplying corresponding elements of the two matrices, and then performing Inverse Fast Fourier Transform (IFFT).

[0060] As a preferred technical measure:

[0061] The following methods can be used to compress the convolutional layers of a neural network:

[0062] Step 21, obtain the weight tensor of the convolutional layer. W The weight tensor of the convolutional layer W For a size of M × N × K d × K r × K c The five-dimensional tensor;

[0063] in, M This represents the number of input channels for the convolutional layer. N This represents the number of output channels of the convolutional layer. K d × K r × K c The kernel size of the convolutional layer;

[0064] Step 22, convert the weight tensorW Divided into K d × K r × K c indivual M × N matrix W ijk ;

[0065] Step 23, with dimensions of M × N matrix W ijk It is constructed as a cyclic matrix to reduce the computational and storage overhead of convolutional layers. The construction method is as follows:

[0066] Will M × N matrix W ijk Divided into p × q indivual B × B submatrix W i,j,kd,kr,kc ,

[0067] in p = M / B , q = N / B , i =0, 1, ..., p -1, j =0, 1, ..., q -1, k d , k r , k c =0, 1, ..., K -1,

[0068] Each submatrix W i,j,kd,kr,kc All are set to a cyclic matrix structure.

[0069] As a preferred technical measure:

[0070] The method to accelerate computation of compressed convolutional layers is as follows:

[0071] S21, Based on the cyclic matrix, the calculation of the convolutional layer is performed according to the block matrix-vector multiplication method;

[0072] S22, the block matrix-vector multiplication is accelerated using Fast Fourier Transform (FFT) to reduce the computational cost of the convolutional layer. To achieve one of the above objectives, the second technical solution of this invention is as follows:

[0073] A three-dimensional neural network-based image processing method is proposed, which uses a three-dimensional neural network processing method based on a block cyclic matrix to process video images in order to extract features in the temporal and spatial dimensions of the video images.

[0074] Through continuous exploration and experimentation, this invention employs a three-dimensional neural network processing method based on block cyclic matrices to process video images. This method enables high-performance and low-power deployment of 3D CNNs, thereby achieving accurate classification and recognition of high-dimensional data such as videos. The solution is scientific, reasonable, practical, and easy to implement.

[0075] To achieve one of the above objectives, the third technical solution of the present invention is as follows:

[0076] A three-dimensional neural network processing system based on a block cyclic matrix includes a block cyclic matrix module, a computation acceleration module, and a full-frequency domain module;

[0077] The block cyclic matrix module uses block cyclic matrices to represent the weights of the neural network, compressing the fully connected layers and / or convolutional layers of the neural network to obtain compressed fully connected layers and / or convolutional layers.

[0078] The computation acceleration module uses Fourier transform to accelerate computation on the compressed fully connected layer and / or convolutional layer to obtain frequency domain computation results;

[0079] The full-frequency domain module selects frequency domain operators and performs activation, batch normalization, and pooling operations on the frequency domain calculation results to complete the three-dimensional neural network processing based on the block cyclic matrix.

[0080] Through continuous exploration and experimentation, this invention overcomes the problems of memory access and computational irregularities inherent in previous module compression methods such as pruning by setting up a block-based cyclic matrix module, a computation acceleration module, and a full-frequency domain module. It proposes using a block-based cyclic matrix to compress 3D neural networks (CNNs) and further utilizes Fast Fourier Transform (FFT) to accelerate computation, achieving significant storage and computational compression effects while maintaining the regularity of the module structure. Building upon this, activation, batch normalization, and pooling operations in the frequency domain are introduced to further eliminate the frequent time-domain / frequency-domain switching overhead caused by FFT, thereby achieving full-frequency domain computation and further reducing the computational overhead during 3D CNN module inference.

[0081] Furthermore, this invention employs a three-dimensional neural network processing system based on a block-based cyclic matrix, which is applicable to various types of convolutional kernel sizes and strides, has a wide range of applications, and possesses strong versatility. In addition, this three-dimensional neural network processing system does not introduce additional computational complexity to the module's training process. The solution is scientifically sound, practical, and easy to implement, effectively addressing the processing efficiency issue of 3D CNNs and enabling its deployment in energy- and resource-constrained scenarios.

[0082] To achieve one of the above objectives, the fourth technical solution of the present invention is as follows:

[0083] A computer-readable storage medium having a computer program stored thereon;

[0084] When the program is executed by the processor, it implements the aforementioned three-dimensional neural network processing method based on a block-based cyclic matrix.

[0085] Or / and, when the program is executed by the processor, it implements the above-described image processing method based on a three-dimensional neural network.

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] Through continuous exploration and experimentation, this invention overcomes the problems of memory access and computational irregularities inherent in previous model compression methods such as pruning by constructing a block-based cyclic matrix model, a computational acceleration model, and a full-frequency domain model. It proposes using a block-based cyclic matrix to compress 3D neural networks (CNNs) and further utilizes Fast Fourier Transform (FFT) to accelerate computation, achieving significant storage and computational compression effects. Building upon this, activation, batch normalization, and pooling operations in the frequency domain are introduced to further eliminate the frequent time-domain / frequency-domain switching overhead caused by FFT, thereby achieving full-frequency domain computation and further reducing the computational overhead during 3D CNN model inference.

[0088] Furthermore, this invention employs a 3D neural network processing method based on block-based cyclic matrices, which is applicable to various types of convolutional kernel sizes and strides, has a wide range of applications, and possesses strong versatility. In addition, this 3D neural network processing method does not introduce additional computational complexity into the model training process. The solution is scientifically sound, practical, and easy to implement, effectively addressing the processing efficiency issue of 3D CNNs and enabling its deployment in energy- and resource-constrained scenarios.

[0089] Furthermore, through continuous exploration and experimentation, this invention employs a three-dimensional neural network processing method based on a block-based cyclic matrix to process video images. This method enables the deployment of 3D CNNs with high performance and low power consumption, thereby achieving accurate classification and recognition of high-dimensional data such as videos. The solution is scientific, reasonable, practical, and easy to implement.

[0090] Furthermore, through continuous exploration and experimentation, this invention overcomes the problems of memory access and computational irregularities inherent in previous module compression methods such as pruning by setting up a block-based cyclic matrix module, a computation acceleration module, and a full-frequency domain module. It proposes using a block-based cyclic matrix to compress 3D neural networks (3D CNNs) and further utilizes Fast Fourier Transform (FFT) to accelerate computation, achieving significant storage and computational compression effects while maintaining the regularity of the module structure. Building upon this, activation, batch normalization, and pooling operations in the frequency domain are introduced to further eliminate the frequent time-domain / frequency-domain switching overhead caused by FFT, thereby achieving full-frequency domain computation and further reducing the computational overhead during 3D CNN module inference. Attached Figure Description

[0091] Figure 1 This is a flowchart of a three-dimensional neural network processing method according to the present invention;

[0092] Figure 2 This is a schematic diagram of the cyclic matrix and its compression method of the present invention;

[0093] Figure 3 This is a schematic diagram illustrating the acceleration of the circular matrix-vector multiplication method using Fast Fourier Transform (FFT) according to the present invention.

[0094] Figure 4 This is a schematic diagram of the block matrix-vector multiplication method for fully connected layers according to the present invention;

[0095] Figure 5 This is a schematic diagram illustrating how the convolutional layer weights are compressed according to the present invention;

[0096] Figure 6 This is a schematic diagram illustrating how the present invention accelerates convolutional layers;

[0097] Figure 7 This is a schematic diagram illustrating the full-frequency domain inference of the present invention;

[0098] Figure 8 This is a schematic diagram illustrating the effect of the compression ratio on accuracy according to the present invention. Detailed Implementation

[0099] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0100] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0101] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the invention. The term "or / and" as used in this application includes any and all combinations of one or more of the associated listed items.

[0102] like Figure 1 As shown, this invention presents a first specific embodiment of a three-dimensional neural network processing method based on a block-based cyclic matrix:

[0103] A three-dimensional neural network processing method based on block cyclic matrices includes the following:

[0104] By using a pre-built block cyclic matrix model, the weights of the neural network are represented by the block cyclic matrix, and the fully connected layers and / or convolutional layers of the neural network are compressed to obtain compressed fully connected layers and / or convolutional layers.

[0105] By using a pre-built computation acceleration model and employing Fourier transform, the compressed fully connected layer and / or convolutional layer are computed at an accelerated speed to obtain frequency domain computation results.

[0106] Based on the pre-constructed full-frequency domain model, frequency domain operators are selected, and frequency domain calculation results are subjected to frequency domain activation, batch normalization, and pooling operations to complete the three-dimensional neural network processing based on the block cyclic matrix.

[0107] A second specific embodiment of the three-dimensional neural network processing method based on block cyclic matrices of the present invention:

[0108] A 3D neural network processing method based on block-based cyclic matrices includes the following:

[0109] A block-based cyclic matrix is ​​used to compress 3D CNNs, and further accelerated computation through Fast Fourier Transform (FFT), effectively reducing the storage and computational overhead of 3D CNNs. Furthermore, activation, batch normalization, and pooling operations in the frequency domain are introduced to eliminate the frequent time-domain / frequency-domain switching overhead caused by FFT.

[0110] To address the massive storage and computational overhead of 3D CNNs, this invention proposes a compression method based on block-based cyclic matrices. This method can be mainly divided into compression and acceleration of fully connected layers of 3D CNNs, compression and acceleration of convolutional layers of 3D CNNs, and the introduction of frequency domain operators to achieve full-frequency domain computation. This effectively alleviates the problem of excessive storage and computational overhead when deploying 3D convolutional neural networks.

[0111] A specific embodiment of the circular matrix of the present invention:

[0112] The circulant matrix of this invention is a special form of the Toeplitz matrix. × matrix It is a cyclic matrix if and only if each element in its row is the result of cyclically shifting the element in the previous row one position to the right, such as... Figure 2 As shown in (a) above, the mathematical formula is: [ ][ ]= [( +1)% ][( +1)% Based on this property, it can be concluded that only the first column elements of matrix A need to be stored (called...). The vector formed by the elements of the first column of the matrix (which is the generating vector of the cyclic matrix) can represent the entire matrix, thus reducing the storage overhead. O ( N 2 Reduced to O ( N Furthermore, circular matrices bring... ( While achieving a compression ratio of 1.5 times, it can also effectively reduce the computational complexity of matrix-vector multiplication: according to the circular convolution theorem, a × Circular matrix And a Any vector ×1 Multiplication can be accelerated using formula (1):

[0113]

[0114] in, It is a circular matrix The generated vectors, Fast Fourier Transform (FFT), and Inverse Fast Fourier Transform (IFFT) represent the Fast Fourier Transform and its inverse transform, respectively. ⊙ represents element-wise multiplication, such as... Figure 2 As shown. Note The computational complexity of pointwise Fast Fourier Transform (FFT) / Inverse Fast Fourier Transform (IFFT) is O(n). ( Therefore, by using the circular convolution theorem, the original... ( 2 The computational complexity is reduced to ( ), thereby obtaining ( / The calculated compression ratio is 1.5 times.

[0115] By definition, a circulant matrix must be a square matrix. However, in most cases, matrices do not possess this form, thus making direct compression using circulant matrices impossible. A simple approach is to first fill the matrix into a square matrix and then compress it using circulant matrices, such as... Figure 2 As shown in (b), the 3×6 matrix is ​​first expanded into a 6×6 square matrix, and then represented as a cyclic matrix, thus achieving compression. However, this method has the following two problems: 1. Filling the matrix into a square matrix incurs additional computational and storage overhead, reducing the benefits of cyclic matrix compression. 2. Representing the entire filled matrix as a cyclic matrix implies a high compression ratio, and this compression ratio cannot be adjusted. Another method addresses this problem by partitioning the matrix. This invention first performs a partitioning operation on the matrix, for example... Figure 2 In (c), the 3×6 matrix is ​​divided into two 3×3 submatrices. Each submatrix is ​​then made to have the structure of a cyclic matrix. Based on the properties of cyclic matrices, it is only necessary to store the generating vector of each cyclic matrix. Figure 2 The compression ratio is achieved by using the 3×2 elements within the red box in (c) of the matrix, resulting in a compression ratio of 3 times (equal to the block size). Clearly, in this compression method, the compression ratio is determined by the block size of the matrix. Therefore, a trade-off between accuracy and compression ratio can be struck by adjusting the block size: a larger block size provides a higher compression ratio, but also results in a greater loss of accuracy, and vice versa.

[0116] Therefore, the compression method based on block-based cyclic matrices adopted in this invention is applicable to various types of convolutional kernel sizes and strides, has a wide range of applications, and possesses strong versatility. Furthermore, this compression method does not introduce additional computational complexity into the model training process.

[0117] The first specific embodiment of this invention for compressing and accelerating fully connected layers of 3D CNN:

[0118] This invention achieves compression by constraining the weights of the fully connected layers of a 3D CNN to block-based cyclic matrices. The input features of the fully connected layers are set... x and output features a The dimensions are respectively N , M Then the weight W It is a size of M × N The matrix. The compression weights. W First, divide it into p × q indivual k × k square array W ij ,in p = M / k , q = N / k Then let each sub-matrix W ij Both are cyclic matrices, thus achieving the goal of compressing the number of parameters, with a corresponding compression ratio of... O ( k After this compression, the computation of the fully connected layer will be reduced. Performed in the form of block matrix multiplication: Then, the Fast Fourier Transform (FFT) is used to multiply each matrix-vector. Accelerate to Through the above optimizations, the computational complexity of the fully connected layer can be reduced from... O ( MN Reduced to O ( pqklok ), obtain O ( k / log k Compression effect of 1.5 times.

[0119] A second specific embodiment of the present invention for compressing and accelerating fully connected layers of 3D CNN:

[0120] In 3D CNNs, a fully connected layer can be represented as a matrix-vector multiplication operation. (Setting input features) and output features The dimensions are respectively , Then the weight W For one × The matrix.

[0121] To compress the fully connected layer using a cyclic matrix, first... × matrix Divided into × indivual × submatrix W ij (here = / , = / If it cannot be divided evenly, then... W Perform padding until divisible, then let each submatrix... W ij Both are cyclic matrices, thus achieving a compression effect. Under this compression strategy, only each submatrix needs to be stored. W ij Generation vector w ij Therefore, the total storage overhead is Compared to the uncompressed state (= 2 The storage overhead reached Compression effect of times.

[0122] Next, consider the weights in the fully connected layer. and input features The calculation, and Similarly, A vector of ×1 Divided into indivual ×1 subvector x j , Will A vector of ×1 Divided into indivual ×1 subvector a i ,but and Multiplication can be done according to Figure 4 The process is divided into blocks as shown, i.e. Note that the weights W It is a block-based cyclic matrix, therefore W ij The given matrix is ​​a cyclic matrix, and its corresponding generating vector is... w ij Therefore, matrix-vector multiplication can be further accelerated by using Fast Fourier Transform (FFT), as shown in formula (2).

[0123]

[0124] To further reduce the number of calls to the Inverse Fast Fourier Transform (IFFT), the linearity of the FFT / IFFT property can be utilized to move the IFFT outside the summation symbol, thereby reducing the number of IFFT calls. q Reduced to 1, that is:

[0125]

[0126] In summary, after block-based cyclic matrix compression, the computational complexity of the fully connected layer can be reduced from O(...). pqk 2 ) reduced to O( pqk log k ).

[0127] Furthermore, unlike compression algorithms such as quantization-aware training and low-rank decomposition, which increase model training complexity, compression based on block cyclic matrices can also accelerate the backpropagation process of the network. The loss function is set as follows: L subvector a i The l The elements are , x j The l The elements are According to the chain rule, we have:

[0128]

[0129]

[0130] It can be proven that and It has a structure of block cyclic matrices, and its generating vectors are respectively =( , , ..., )andw ij .therefore, and The computational complexity of the model training phase can also be accelerated by using Fast Fourier Transform (FFT) --> Element-WiseMultiplication --> Inverse Fast Fourier Transform (IFFT). Algorithm 1 demonstrates the computation process of backpropagation in the fully connected layer after block cyclic matrix compression.

[0131] Algorithm 1: Backpropagation algorithm for fully connected layers after block cyclic matrix compression

[0132]

[0133] A specific embodiment of the present invention for compressing and accelerating 3D CNN convolutional layers:

[0134] Unlike 2D CNNs, convolutions in 3D CNNs add an extra time dimension. Therefore, in 3D CNNs, the weights of the convolutional layers... W It is like Figure 5 The five-dimensional tensor shown in (a) is W R M×N×K×K×K .in, M , N , K These represent the number of output channels, the number of input channels, and the kernel size of the convolutional layer, respectively. Considering the kernel size... K Since the values ​​are often small, this application chooses to explore the cyclic matrix structure in both the input and output channels.

[0135] like Figure 5 As shown in (b) above, firstly... M × N × K × K × K The weight tensor is regarded as K 3 indivual M × N The matrix, then, each M × N The matrix is ​​divided into p × q indivual B × B submatrix W i,j,kd,kr,kc ,in p = M / B , q = N / B , i =0, 1, ..., p -1, j =0, 1, ..., q -1, k d , k r , k c =0, 1, ..., K -1, and finally, let each submatrix... W i,j,kd,kr,kc They all have a cyclic matrix structure.

[0136] With the compression method described above, only each submatrix needs to be stored. W i,j,kd,kr,kc Generation vector w i,j,kd,kr,kc ,like Figure 5 As shown in (c), this reduces the storage overhead from the original MNK 3 Reduce to the present pqBK 3 , obtain B Compression effect of times.

[0137] Further consideration is given to the computation of convolutional layers after block-based cyclic matrix compression, such as... Figure 6 As shown, ① is the pseudocode for 3D convolution. According to ①, for a given... d , r , c , k d , k r , k c , O [ pB + B : pB ][ d ][ r ][ c ]、 W [ pB + B : pB ][ qB + B : qB ][ k d ][ k r ][ k c ]、I[ qB + B : qB ][d + k d ][ r + k r ][ c + k c The dimensions are respectively M ×1、 M × N , N The operations between tensors of size ×1 are essentially matrix-vector multiplications. Therefore, 3D convolution can be viewed as a series of matrix-vector multiplication operations.

[0138] Since the convolutional layer weights have already undergone block-based cyclic matrix compression, ① can be further rewritten as the block matrix-vector multiplication form in ②. And for each given... p , q , k d , k r , k c , W [ pB + B : pB ][ qB + B : qB ][ k d ][ k r ][ k c Both have the structure of a circular matrix; therefore, ② can be further optimized into ③ using Fast Fourier Transform (FFT), where... w [ p ][ q ][:][ k d ][ k r ][ k c ] is a circular matrix W [ pB + B : pB ][ qB + B : qB ][ k d ][ k r ][ k c The corresponding generated vector.

[0139] Thanks to various forms of data reuse in 3D convolution, block cyclic matrix compression and Fast Fourier Transform (FFT) acceleration bring even more significant benefits to convolutional layers. According to Figure 6 As shown in ③, the computational complexity of performing Fast Fourier Transform (FFT) / Inverse Fast Fourier Transform (IFFT) on the input feature map, weights, and output feature map are respectively... O ( NDRC log B ), O ( K 3 pqB log B )and O ( MDRC log B The computational complexity of element-wise multiplication is O(n). O ( MNDRCK 3 / B Since the computational complexity of an uncompressed convolutional layer is... O ( MNDRCK 3 Therefore, the theoretical compression ratio of computational complexity can be described by formula (6):

[0140]

[0141] Considering that in 3D CNN, , , , , The value is much greater than Therefore, the computational complexity of the compressed convolutional layer (i.e., the denominator in formula (6)) is approximately: ( 3 / B The corresponding compression ratio is approximately ( B Compared to fully connected layers ( B / log B The compression effect is 1.5 times that of the previous method, resulting in a significant improvement.

[0142] A specific embodiment of the present invention for full-frequency domain calculation:

[0143] Because the computation of the cyclic matrix is ​​accelerated by the Fast Fourier Transform (FFT), the computation process of convolutional and fully connected layers becomes FFT → EWM → Inverse Fast Fourier Transform (IFFT), where EWM is element-wise multiplication. However, the computation of activation function layers, batch normalization layers, and pooling layers all need to be performed in the time domain. Therefore, frequent time-domain / frequency-domain switching overhead (FFT / IFFT) occurs during the forward inference process of the entire model. This invention eliminates this overhead by introducing activation, batch normalization, and pooling operations in the frequency domain, thereby further reducing the computational load during model inference.

[0144] This invention introduces frequency domain operators to achieve full-frequency domain computation in a specific embodiment:

[0145] After compressing the convolutional and fully connected layers of a 3D CNN using a block-based cyclic matrix and accelerating them with Fast Fourier Transform (FFT), the computation flow of the convolutional and fully connected layers is reduced to FFT. EWM Inverse Fast Fourier Transform (IFFT)

[0146] However, this raises another problem: such as Figure 7 As shown in (a), since activation, batch normalization, and pooling layers are all calculated in the time domain, after each calculation of a convolutional or fully connected layer in the frequency domain, it is necessary to convert it to the time domain using Inverse Fast Fourier Transform (IFFT) to perform activation, pooling, batch normalization, and other operations. After the calculation is completed, if the next convolution or fully connected operation is to be performed, the features must be converted to the frequency domain using Fast Fourier Transform (FFT).

[0147] Therefore, while accelerating convolutional / fully connected layers using Fast Fourier Transform (FFT) significantly reduces computation, it also introduces frequent time-domain / frequency-domain switching overhead (FFT / IFFT) during network inference. This switching overhead leads to three problems: 1. FFT / IFFT are computationally intensive algorithms, with the conversion process itself involving significant computation; 2. Designing hardware accelerators requires dedicated hardware modules to compute FFT / IFFT, consuming additional hardware resources; 3. FFT / IFFT operations introduce computational errors, which are unfriendly to model quantization.

[0148] To address the aforementioned problems, this invention proposes introducing activation, batch normalization, and pooling operations from the time domain into the frequency domain to achieve full-frequency domain computation, thereby eliminating the overhead of frequent time-domain / frequency-domain switching. Figure 7 As shown in (b) of the diagram.

[0149] When selecting frequency domain operators, two main aspects are considered: firstly, the computational complexity of the frequency domain operators must not be too high, otherwise the additional computational load brought by the frequency domain operators will offset the benefits brought by the full frequency domain computation; secondly, the introduction of frequency domain operators should not cause a significant loss of accuracy. Balancing these two points, this invention uses CReLU, CBN, and CMaxPool as frequency domain operators, and their expressions are shown in formulas (7), (8), and (9), respectively.

[0150]

[0151]

[0152]

[0153] ReLU, BN, and MaxPool represent activation, batch normalization, and max pooling operations in the real domain, respectively. Table 1 lists the ablation experiments of these frequency domain operators on the C3D network model and the UCF101 dataset. As can be seen from the table, replacing ReLU with CReLU only resulted in a 0.37% loss in accuracy, while replacing BN and Maxpool with CBN and CMaxpool respectively improved accuracy by 0.556% and 0.417%.

[0154] Table 1. Ablation Experiments Using Frequency Domain Operators in C3D

[0155]

[0156] An image processing embodiment applying the method of the present invention:

[0157] A three-dimensional neural network-based image processing method is proposed, which uses a three-dimensional neural network processing method based on a block cyclic matrix to process video images in order to extract features in the temporal and spatial dimensions of the video images.

[0158] A system embodiment applying the method of the present invention:

[0159] A three-dimensional neural network processing system based on a block cyclic matrix includes a block cyclic matrix module, a computation acceleration module, and a full-frequency domain module;

[0160] The block cyclic matrix module uses block cyclic matrices to represent the weights of the neural network, compressing the fully connected layers and / or convolutional layers of the neural network to obtain compressed fully connected layers and / or convolutional layers.

[0161] The computation acceleration module uses Fourier transform to accelerate computation on the compressed fully connected layer and / or convolutional layer to obtain frequency domain computation results;

[0162] The full-frequency domain module selects frequency domain operators and performs activation, batch normalization, and pooling operations on the frequency domain calculation results to complete the three-dimensional neural network processing based on the block cyclic matrix.

[0163] An embodiment of a device applying the method of the present invention:

[0164] A computer device comprising:

[0165] One or more processors;

[0166] Storage device for storing one or more programs;

[0167] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described three-dimensional neural network processing method based on a block-based cyclic matrix.

[0168] An embodiment of a computer medium applying the method of the present invention:

[0169] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described three-dimensional neural network processing method based on a block cyclic matrix.

[0170] A specific embodiment for experimental verification of the present invention:

[0171] To evaluate this invention, two 3D CNN models, C3D and 3D ResNet-18, were selected for experiments. C3D has a very regular network structure, consisting of 8 convolutional layers, 5 pooling layers, 2 fully connected layers, and 1 softmax layer. All convolutional kernels are 3×3×3 in size with a stride of 1×1×1. All max-pooling layers, except for the first pooling kernel which has a size and stride of 1×2×2, are 2×2×2.

[0172] 3D ResNet-18 evolved from 2D ResNet-18. Therefore, 3D ResNet-18 includes not only standard 3×3×3 convolutions but also 1×1×1 point-wise convolutions. Furthermore, 3D ResNet-18 features residual connections to address the vanishing and exploding gradient problems in deep networks. The experimental dataset used is UCF101, which includes 13,320 videos across 101 categories at a resolution of 320×240, with a variable frame rate, typically 25 or 29 frames per second. In the experiments, all ReLU, BN, and Maxpool operations were replaced with CReLU, CBN, and CMaxpool to achieve full-frequency domain computation. Additionally, INT8 quantization-aware training was performed during training. Throughout the training process, the hyperparameter settings for the experiment were as follows: the spatial size of the input video was 112×112, the number of channels was 3, the total number of frames was 16, the initial learning rate was 0.01, the cosine learning rate update strategy was adopted, the training was conducted for 300 epochs, and the batch sizes for C3D and 3D ResNet-18 were 64 and 128, respectively.

[0173] To investigate the impact of block size in the block-based cyclic matrix on model accuracy, experiments were conducted with block sizes of 1, 4, 8, and 16, respectively. The results are as follows: Figure 8 As shown in the figure, when the block size is 4, the accuracy loss of C3D and 3D ResNet-18 is only 0.278% and 0.51%, respectively, with corresponding storage and computation compression ratios of 16 times and 2.67 times. In contrast, Sun et al. pruned C3D on a block-by-block basis, achieving a parameter compression ratio of 1.05 times and a computation compression ratio of 3.18 times at the cost of a 0.34% accuracy loss. Clearly, this invention achieves a more significant parameter compression effect with a smaller accuracy loss (0.278%). Regarding computational compression, although the compression effect achieved by this invention is 1.19 times lower than that of Sun et al., the method proposed in this invention provides highly regular compression of C3D, thus achieving better results in practical hardware deployment.

[0174] When the block size is 16, the accuracy loss of C3D and 3D ResNet-18 is 2.825% and 3.196%, respectively, indicating that compression has a significant impact on accuracy. Therefore, in order to achieve a better trade-off between accuracy and compression ratio, the cyclic matrix block size is ultimately set to 8 to compress C3D and 3D ResNet-18, with corresponding storage and computation compression ratios of 32 times and 5.33 times, respectively, and accuracy losses of 1.25% and 1.71%, respectively.

[0175] In contrast, the working 3D VNPU[2] uses the Winograd algorithm and INT8 quantization, achieving a 4x storage compression ratio and a 3.33x computation compression ratio at the cost of a 1% accuracy loss. Through analysis, it is not difficult to find that, with a slightly larger accuracy loss (0.25%), this invention further improves the storage and computation compression ratios by 8x and 1.6x respectively on the basis of 3D VNPU, which fully demonstrates the superiority of this invention in 3D CNN model compression.

[0176] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0177] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0178] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0179] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A three-dimensional neural network processing method based on a block-based cyclic matrix, characterized in that, Includes the following: By using a pre-built block cyclic matrix model, the weights of the neural network are represented by the block cyclic matrix, and the fully connected layers and / or convolutional layers of the neural network are compressed to obtain compressed fully connected layers and / or convolutional layers. By using a pre-built computation acceleration model and employing Fourier transform, the compressed fully connected layer and / or convolutional layer are computed at an accelerated speed to obtain frequency domain computation results. Based on the pre-built full-frequency domain model, frequency domain operators are selected, and frequency domain calculation results are subjected to frequency domain activation, batch normalization, and pooling operations to complete the three-dimensional neural network processing based on the block cyclic matrix. The method for constructing a block-based cyclic matrix model is as follows: Step 1, settings × matrix , is used to represent a circular matrix; The second step is to construct the elements of matrix A if and only if each row of elements is the result of the previous row of elements cyclically shifted one position to the right. The entire matrix A is represented by storing the elements of the first column of matrix A, and the vector formed by the elements of the first column is the generating vector of the cyclic matrix. The method for constructing the computational acceleration model is as follows: Based on the circular convolution algorithm, the multiplication of circular matrices and arbitrary vectors is accelerated; The circular matrix is × matrix ; Vector is A vector of ×1 ; The calculation formula for the circular convolution algorithm is as follows: in, It is a matrix The generated vector, FFT is the Fast Fourier Transform, IFFT is the Inverse Fast Fourier Transform, and ⊙ represents element-wise multiplication; The method for constructing the full-frequency domain model is as follows: Based on the computational complexity and accuracy loss of frequency domain operators, CReLU, CBN, and CMaxPool are adopted as frequency domain operators, and their expressions are shown below: ReLU, BN, and MaxPool are the activation, batch normalization, and max pooling operations on the real number field, respectively. a b is the value in the real number field, and b is the value in the imaginary number field.

2. The three-dimensional neural network processing method based on a block-based cyclic matrix as described in claim 1, characterized in that, The method for compressing the fully connected layers of a neural network is as follows: Step 11, obtain the weight matrix of the fully connected layer. W Weight matrix W The size is M × N ; Step 12, with dimensions of M × N weight matrix W Divided into p × q indivual k × k square array W ij ; = / , = / If it cannot be divided evenly, then... W Continue filling until the result is divisible; Step 13, arrange the square array W ij It is constructed as a cyclic matrix to compress the number of parameters in a neural network.

3. The three-dimensional neural network processing method based on a block-based cyclic matrix as described in claim 2, characterized in that, The method for accelerating computation of the compressed fully connected layer is as follows: S11, based on the cyclic matrix, calculate the fully connected layer. a = W x The calculation is performed using block matrix-vector multiplication, and the formula is as follows: ， in, a To output feature vectors ，W It is a block-based cyclic matrix. The input feature vector; This is a subvector of the output feature vector. W ij It is a circular matrix. A subvector of the input feature vector; S12 accelerates the block matrix-vector multiplication using Fast Fourier Transform (FFT) to reduce the computational cost of the fully connected layer. The calculation formula is as follows: S13, utilizing the linearity of Fast Fourier Transform (FFT) / Inverse Fast Fourier Transform (IFFT), moves the Inverse Fast Fourier Transform (IFFT) outside the summation symbol, reducing the number of IFFT calls from... q The formula for reducing it to 1 is as follows: Here, FFT stands for Fast Fourier Transform, and IFFT stands for Inverse Fast Fourier Transform.

4. The three-dimensional neural network processing method based on a block-based cyclic matrix as described in claim 3, characterized in that, It also includes accelerating the backpropagation process of the neural network using a block-based cyclic matrix, the expression of which is as follows: in, L For loss function, Subvectors of the output feature vector a i The l One element, Subvectors of the input feature vector x j The l One element, The generating vector corresponding to the circular matrix is w ij ; and It has a structure with block cyclic matrices, and its generating vectors are respectively =( , , ..., )and w ij ; Will and The process is accelerated by sequentially performing Fast Fourier Transform (FFT), multiplying corresponding elements of the two matrices, and then performing Inverse Fast Fourier Transform (IFFT).

5. A three-dimensional neural network processing method based on a block-based cyclic matrix as described in any one of claims 1-4, characterized in that, The following methods can be used to compress the convolutional layers of a neural network: Step 21, obtain the weight tensor of the convolutional layer. W The weight tensor of the convolutional layer W For a size of M × N × K d × K r × K c The five-dimensional tensor; in, M This represents the number of input channels for the convolutional layer. N This represents the number of output channels of the convolutional layer. K d × K r × K c The kernel size of the convolutional layer; Step 22, convert the weight tensor W Divided into K d × K r × K c indivual M × N matrix W ijk ; Step 23, with dimensions of M × N matrix W ijk It is constructed as a cyclic matrix to reduce the computational and storage overhead of convolutional layers. The construction method is as follows: Will M × N matrix W ijk Divided into p × q indivual B × B submatrix W i,j,kd,kr,kc , in p = M / B , q = N / B , i =0, 1, ..., p -1, j =0, 1, ..., q -1, k d , k r , k c =0, 1, ..., K -1, Each submatrix W i,j,kd,kr,kc All are set to a cyclic matrix structure.

6. The three-dimensional neural network processing method based on a block-based cyclic matrix as described in claim 5, characterized in that, The method to accelerate computation of compressed convolutional layers is as follows: S21, Based on the cyclic matrix, the calculation of the convolutional layer is performed according to the block matrix-vector multiplication method; S22 accelerates block matrix-vector multiplication using Fast Fourier Transform (FFT) to reduce the computational cost of convolutional layers.

7. An image processing method based on a three-dimensional neural network, characterized in that, The video image is processed using a three-dimensional neural network processing method based on a block cyclic matrix as described in any one of claims 1-6, in order to extract features in the temporal and spatial dimensions of the video image.

8. A three-dimensional neural network processing system based on a block-based cyclic matrix, characterized in that, It employs a three-dimensional neural network processing method based on a block-based cyclic matrix as described in any one of claims 1-6; The three-dimensional neural network processing system based on block cyclic matrix includes a block cyclic matrix module, a computation acceleration module, and a full-frequency domain module; The block cyclic matrix module uses block cyclic matrices to represent the weights of the neural network, compressing the fully connected layers and / or convolutional layers of the neural network to obtain compressed fully connected layers and / or convolutional layers. The computation acceleration module uses Fourier transform to accelerate computation on the compressed fully connected layer and / or convolutional layer to obtain frequency domain computation results; The full-frequency domain module selects frequency domain operators and performs activation, batch normalization, and pooling operations on the frequency domain calculation results to complete the three-dimensional neural network processing based on the block cyclic matrix.

9. A computer-readable storage medium, characterized in that, It contains computer programs; When executed by the processor, the program implements a three-dimensional neural network processing method based on a block-based cyclic matrix as described in any one of claims 1-6; Or / and, when the program is executed by the processor, it implements the image processing method based on a three-dimensional neural network as described in claim 7.

Images