A convolutional neural network acceleration method and a convolutional neural network accelerator

Abstract

The application discloses a convolutional neural network acceleration method and a convolutional neural network accelerator, and belongs to the technical field of computers. The convolutional neural network acceleration method comprises the following steps: obtaining model parameters of a pre-trained convolutional neural network, wherein the model parameters comprise convolution kernel parameters in each convolution layer; obtaining the average value of the central symmetric elements of each layer of convolution kernel, and replacing the average value with the parameters of the convolution kernel in each convolution layer; given an input / output sample pair, training the convolutional neural network based on the input / output sample pair, and ending the training process when a specified error range or a maximum number of times is reached. The processing mode of the convolution kernel center adopted by the application reduces the space required for weight storage and the access frequency of external memory, so that the acceleration of the convolutional neural network has the purpose of high operation efficiency and less required storage space.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to a method for accelerating convolutional neural networks and a convolutional neural network accelerator. Background Technology

[0002] Convolutional Neural Networks (CNNs), as a typical feedforward deep neural network, have been widely used in fields such as text recognition, image recognition, object tracking, and face detection. Improvements in CNN performance are typically associated with increased depth and number of parameters, leading to excessive computation and memory access. Given the current trend towards deeper and larger CNN models to achieve higher accuracy, efficiently handling large-scale CNNs remains a significant challenge.

[0003] Convolutional neural network (CNN) accelerator design mainly focuses on two aspects: First, constructing sparse network architectures through pruning and other methods. Sparsity has become an effective way to reduce data and computation in CNNs. The paper "Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding" proposes a network sparsification method, which mainly involves pruning weights below a threshold to zero and then retraining to maintain the original accuracy, thereby significantly reducing data access and computation. However, the problem with this method is that sparsity leads to irregular computation, which in turn affects the performance of dense accelerators during computation. Second, using structured matrix representations to reduce weight storage costs. The paper "CirCNN: Accelerating and compressing deep neural networks using block-circulant weight matrices" uses block circulant matrices to represent the network. The problem with this method is that it requires complex hardware and involves complex number operations, which are much more expensive than real number operations. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and accelerator for accelerating convolutional neural networks.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] According to a first aspect of the present invention, a method for accelerating convolutional neural networks includes:

[0007] S100. Obtain the model parameters of the pre-trained convolutional neural network, wherein the model parameters include the kernel parameters in each convolutional layer;

[0008] S200. Calculate the average value of the central symmetric elements of the convolution kernel in each convolutional layer based on the kernel parameters in each convolutional layer, and replace it with the kernel parameters in each convolutional layer;

[0009] S300. Given an input / output sample pair, train the convolutional neural network based on the input / output sample pair, and end the training process when the specified error range or the maximum number of iterations is reached.

[0010] Further, S300 includes the following steps:

[0011] S310. Given an input / output sample pair;

[0012] S320. Determine whether the convolutional neural network has reached the maximum number of training iterations. If yes, end the training process; otherwise, execute S330.

[0013] S330. Use the input sample pairs as input vectors to propagate the convolutional neural network forward and calculate the output of the hidden layer units one by one;

[0014] S340. Use the output of the hidden layer unit obtained in S330 as the input vector to calculate the output of the output unit;

[0015] S350. Calculate the error of the output unit;

[0016] S360. Determine whether the error of the output unit meets the preset condition. If yes, end the training process; otherwise, execute S370.

[0017] S370. Calculate the error of the hidden layer elements;

[0018] S380. Determine whether the hidden layer unit of the current calculation error is a convolutional layer. If yes, calculate the center-symmetric error gradient; otherwise, calculate the error gradient.

[0019] S390. Adjust the weights based on the gradient convolution kernel calculated in S380, and then execute S320.

[0020] Furthermore, the error of the output unit is the difference between the output of the output unit and the given output sample pair.

[0021] Furthermore, the error of the hidden layer unit is calculated, including:

[0022] Based on the error of the output unit, the error of the hidden layer unit is calculated according to the chain rule.

[0023] According to a second aspect of the present invention, a convolutional neural network accelerator includes:

[0024] Global cache module;

[0025] Multiple computing engines, which are used to perform convolution operations;

[0026] The control unit is used to read the weight data and bias data required for calculation from the external DRAM, cache the weight data in the weight cache of the global cache module, cache the bias data in the weight cache of the global cache module, cache the given input pair in the input feature cache of the global cache module; and control the computing engine to read the weight data, bias data and input features in the global cache module, perform convolution operation in the computing engine unit to obtain output features, cache the output features in the output feature cache of the global cache module, and write the output features in the output feature cache back to the external DRAM memory.

[0027] Furthermore, the computing engine includes:

[0028] The weight encoding unit is used to receive weight data from the weight buffer and perform sparse encoding on the weight data, selecting Px weight vectors to represent the weight data, thus obtaining sparse weights.

[0029] The input feature encoding unit is used to accept the input features from the input feature buffer and perform sparse encoding on the input features, selecting Py input activation vectors to represent the input features, thus obtaining sparse input features;

[0030] A multiplier array is used to accept Px weight vectors from sparse weights and Py input activation vectors from sparse input features, then perform a Cartesian product of the weight vectors and the input activation vectors, and generate the multiplier array output.

[0031] The coordinate calculation unit is used to accept the weight index provided by the weight encoding unit and the input feature index provided by the input feature encoding unit, and calculate the coordinates of the multiplier output based on the weight index and the input feature index;

[0032] The addressing unit is used to receive the Cartesian product and the coordinates output by the multiplier, and to add the Cartesian product to the corresponding address in the cache spaces 0 and 1 according to the coordinates output by the multiplier, wherein cache spaces 0 and 1 refer to the accumulator caches 0 and 1 inside the accelerator.

[0033] The nonlinear operation unit is used to accept the output data of buffer space 0 and 1, perform nonlinear operations on the output data of buffer space 0 and 1, and output the result of the nonlinear operation to the output feature buffer for caching.

[0034] Furthermore, the weight index is generated as follows:

[0035] Calculate the number of non-zero elements in the weighted data;

[0036] Use the number of non-zero elements in the weight data as the first element of the weight index;

[0037] Calculate the number of zero elements between non-zero values ​​and adjacent non-zero values ​​in the weights;

[0038] The number of zero elements between non-zero values ​​and adjacent non-zero values ​​in the weights are used as weight indices for the other elements except the first element.

[0039] Furthermore, the sparse weights store non-zero weights in a sequential order.

[0040] Furthermore, the input feature index is generated as follows:

[0041] Calculate the number of non-zero elements in the input features;

[0042] The number of non-zero elements in the input features is used as the first element of the input feature index.

[0043] Calculate the number of zero elements between non-zero values ​​and adjacent non-zero values ​​in the input features;

[0044] The number of zero elements between non-zero values ​​and adjacent non-zero values ​​in the input features are used as the input feature indices for the other elements except the first element.

[0045] Furthermore, sparse input features store non-zero input feature elements.

[0046] The beneficial effects of this invention are as follows: This invention provides a method for optimizing convolutional kernels, namely, a neural network acceleration method for centrally processing convolutional kernels, used to accelerate computationally intensive and storage-intensive deep learning networks. Throughout the iterative computation process, the central symmetry of the convolutional kernel elements is maintained by averaging the backpropagation error of elements at centrally symmetric positions. Based on this, a convolutional neural network accelerator suitable for centrally symmetric processing of convolutional kernels is designed. This accelerator mainly consists of external DRAM, a control unit, a global cache, and multiple computing engines. The convolutional kernel centralization method adopted in this invention reduces the space required for weight storage and the number of external memory accesses, enabling the acceleration of convolutional neural networks to achieve high computational efficiency with less storage space requirements. Attached Figure Description

[0047] Figure 1 This is a flowchart of one embodiment of the convolutional neural network acceleration method in this invention;

[0048] Figure 2 This is a schematic diagram of an element before and after kernel centering of a convolutional layer.

[0049] Figure 3 A flowchart of one embodiment of kernel centering processing;

[0050] Figure 4 This is a schematic diagram of an overall architecture of the convolutional neural network accelerator in this invention;

[0051] Figure 5 This is a schematic diagram of a computing engine structure;

[0052] Figure 6 This is a schematic diagram of an example operation of a convolutional neural network accelerator. Detailed Implementation

[0053] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] See Figures 1-6 This embodiment provides a method for accelerating convolutional neural networks and a convolutional neural network accelerator:

[0055] The first aspect of this invention provides a method for accelerating convolutional neural networks. For example... Figure 1 As shown, the method includes steps S100-S300, which are described in detail below.

[0056] Step S100. Obtain the model parameters of the pre-trained convolutional neural network, wherein the model parameters include the convolution kernel parameters in each convolutional layer.

[0057] Specifically, the model parameters include the convolution kernel parameters in each convolutional layer and the parameters of the fully connected layers. The purpose of obtaining the model parameters is to facilitate the initialization of the convolution kernels before the kernel centering process and to facilitate retraining of the network.

[0058] Step S200. Calculate the average value of the center-symmetric elements of the convolution kernel in each convolutional layer based on the kernel parameters in each convolutional layer, and replace it with the kernel parameters in each convolutional layer.

[0059] Specifically, before the entire convolutional kernel centering process begins, the elements of the pre-trained convolutional kernel must first be initialized. This initialization is done by replacing the original values ​​with the mean of the centrally symmetric elements within the kernel, such as... Figure 2 As shown.

[0060] Step S300. Given an input / output sample pair, train the convolutional neural network based on the input / output sample pair, and end the training process when the specified error range or the maximum number of iterations is reached.

[0061] In the process of kernel centering, if only the elements of the kernel are initialized, the performance of the entire convolutional neural network will be significantly reduced. Therefore, in this embodiment, given input / input sample pairs, the convolutional neural network is retrained based on the input / output sample pairs, and the training process ends when the specified error range or the maximum number of iterations is reached.

[0062] In some embodiments, such as Figure 3 As shown, S300 includes the following steps:

[0063] Step S310. Given an input / output sample pair.

[0064] Step S320. Determine whether the convolutional neural network has reached the maximum number of training iterations. If yes, end the training process; otherwise, execute S330.

[0065] Step S330. Use the input sample pairs as input vectors to propagate the convolutional neural network forward and calculate the output of the hidden layer units one by one.

[0066] Calculating the output of the hidden layer units is the forward propagation process of a convolutional neural network. By successively calculating the output of each hidden layer, the output of the output layer can be obtained, and the loss function can be calculated, thus completing the backpropagation process of the error.

[0067] Step S340. Use the output of the hidden layer unit obtained in S330 as the input vector to calculate the output of the output unit.

[0068] The output of the calculation unit is mainly calculated by the forward process based on the input, and the error function value is calculated based on the obtained output of the output unit and the given label.

[0069] Step S350. Calculate the error of the output unit.

[0070] The error of the output unit is calculated mainly by calculating the error between the output of the output unit and the given output sample pair, which facilitates the updating of parameters in the network through error backpropagation.

[0071] Step S360. Determine whether the error of the output unit meets the preset condition. If yes, end the training process; otherwise, proceed to step S370. After the training process ends, save all parameters of the convolutional neural network for convenient use in a hardware accelerator.

[0072] Step S370. Calculate the error of the hidden layer elements.

[0073] Specifically, the calculation of hidden layer unit errors includes: calculating hidden layer unit errors based on the output unit errors according to the chain rule.

[0074] Step S380. Determine whether the hidden layer unit of the current calculation error is a convolutional layer. If yes, calculate the center-symmetric error gradient; otherwise, calculate the error gradient.

[0075] Specifically, if the current layer is a convolutional layer, the centrally symmetric error gradient is calculated, which is done by averaging the gradients during the backpropagation of the error. If the current layer is a non-convolutional layer, the gradient is calculated using the normal gradient calculation method.

[0076] Step S390. Adjust the weights of the convolution kernel according to the gradient calculated in step S380, and then execute step S320.

[0077] A second aspect of the present invention provides a convolutional neural network accelerator. For example... Figure 4 As shown, the convolutional neural network accelerator includes a control unit, a global cache module, and multiple computing engines.

[0078] The control unit is used to read the weight data and bias data required for computation from external DRAM. The bias data is used for convolution computation. The control unit caches the weight data and bias data in the weight cache of the global cache module, and caches the given input pairs in the input feature cache of the global cache module. The control unit also controls the computation engine to read the weight data, bias data, and input features from the global cache module, perform convolution operations in the computation engine unit to obtain output features, cache the output features in the output feature cache of the global cache module, and write the output features from the output feature cache back to the external DRAM memory.

[0079] External DRAM is primarily used to store the weights, biases, and input feature data required by convolutional neural networks. In convolutional neural networks, the weights, biases, and input feature data are often quite large, requiring significant storage space. Therefore, on-chip memory is often insufficient to meet these storage needs, necessitating the use of off-chip memory.

[0080] The computation engine is used to perform convolution operations. Specifically, such as... Figure 5 As shown, the computing engine includes a weight encoding unit, an input feature encoding unit, a multiplier array, a coordinate calculation unit, an addressing unit, and a nonlinear operation unit.

[0081] The weight encoding unit receives weight data from the weight cache and performs sparse encoding on the weight data, thereby selecting Px weight vectors to represent the weight data more efficiently, thus obtaining sparse weights. In some embodiments, the weight index is generated as follows: the number of non-zero elements in the weight data is calculated; the number of non-zero elements in the weight data is used as the first element of the weight index; the number of zero elements between non-zero values ​​and adjacent non-zero values ​​in the weights is calculated; the number of zero elements between non-zero values ​​and adjacent non-zero values ​​in the weights is used as the other elements of the weight index, excluding the first element. The sparse weights store non-zero weights in a sequentially ordered manner.

[0082] The input feature encoding unit accepts input features from the input feature buffer and performs sparse encoding on the input features, thereby selecting Py input activation vectors to more efficiently represent the input features, i.e., obtaining sparse input features. In some embodiments, the input feature index is generated as follows: calculate the number of non-zero elements in the input features; use the number of non-zero elements in the input features as the first element of the input feature index; calculate the number of zero elements between non-zero values ​​and adjacent non-zero values ​​in the input features; and use the number of zero elements between non-zero values ​​and adjacent non-zero values ​​in the input features as the other elements of the input feature index, excluding the first element. The sparse input features store non-zero input feature elements.

[0083] The multiplier array is used to accept Px weight vectors from sparse weights and Py input activation vectors from sparse input features, then perform a Cartesian product of the weight vectors and the input activation vectors, and generate the multiplier array output.

[0084] The coordinate calculation unit is used to accept the weight index provided by the weight encoding unit and the input feature index provided by the input feature encoding unit, and calculate the coordinates of the multiplier output based on the weight index and the input feature index.

[0085] The addressing unit is used to receive the Cartesian product and the coordinates output by the multiplier, and to add the Cartesian product to the corresponding address in the cache spaces 0 and 1 according to the coordinates output by the multiplier, wherein cache spaces 0 and 1 refer to the accelerator's internal accumulator caches 0 and 1.

[0086] The nonlinear operation unit is used to accept the output data of buffer space 0 and 1, perform nonlinear operations on the output data of buffer space 0 and 1, and output the result of the nonlinear operation to the output feature buffer for caching.

[0087] Figure 6The specific steps for a convolutional neural network accelerator computation example are as follows: First, the pre-trained convolutional neural network model parameters are obtained, and the kernel parameters are optimized using the mean of the centrosymmetric elements of the convolution kernel. Then, the weights, bias data, and the given input-output pairs are stored in external DRAM. Next, the control unit caches the above data in the global cache module and further controls the computing engine to read the above data. After sparse encoding, sparse weights, weight indices, sparse input features, and input feature indices are obtained. Then, the multiplier array receives Px weight vectors from the sparse weights and Py input activation vectors from the sparse input features, and performs a Cartesian product of the weight vectors and input activation vectors to generate the multiplier array output. At the same time, the coordinate calculation unit receives the weight index and the input feature index, and calculates the coordinates of the multiplier output based on the weight index and the input feature index. Then, these two outputs are received by the addressing unit, and the Cartesian product is accumulated into the corresponding addresses in the cache spaces 0 and 1 according to the coordinates of the multiplier output, i.e., accumulator caches 0 and 1. Finally, the output data from buffer spaces 0 and 1 will be transmitted to the nonlinear operation unit for nonlinear operation, and the result will be output to the output feature buffer for caching.

[0088] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A convolutional neural network accelerator, characterized in that, include: Global cache module; Multiple computing engines, which are used to perform convolution operations; The control unit is used to read the weight data required for calculation and the bias data used for convolution calculation from the external DRAM, and cache the weight data in the weight cache of the global cache module, cache the bias data in the weight cache of the global cache module, and cache the given input pair in the input feature cache of the global cache module. And for controlling the computing engine to read weight data, bias data and input features from the global cache module, and to perform convolution operation in the computing engine unit to obtain output features, and to cache the output features in the output feature cache in the global cache module, and to write the output features in the output feature cache back to the external DRAM memory; The computing engine includes: The weight encoding unit is used to receive weight data from the weight buffer and perform sparse encoding on the weight data, selecting Px weight vectors to represent the weight data, thus obtaining sparse weights. The input feature encoding unit is used to accept the input features from the input feature buffer and perform sparse encoding on the input features, selecting Py input activation vectors to represent the input features, thus obtaining sparse input features; A multiplier array is used to accept Px weight vectors from sparse weights and Py input activation vectors from sparse input features, then perform a Cartesian product of the weight vectors and the input activation vectors, and generate the multiplier array output. The coordinate calculation unit is used to accept the weight index provided by the weight encoding unit and the input feature index provided by the input feature encoding unit, and calculate the coordinates of the multiplier output based on the weight index and the input feature index; The addressing unit is used to receive the Cartesian product and the coordinates output by the multiplier, and to add the Cartesian product to the corresponding address in the cache space 0 and 1 according to the coordinates output by the multiplier, wherein cache space 0 and 1 refer to the accelerator's internal accumulator cache 0 and 1. The nonlinear operation unit is used to accept the output data of buffer space 0 and 1, perform nonlinear operations on the output data of buffer space 0 and 1, and output the result of the nonlinear operation to the output feature buffer for caching.

2. The convolutional neural network accelerator according to claim 1, characterized in that, The weight index is generated as follows: Calculate the number of non-zero elements in the weighted data; Use the number of non-zero elements in the weight data as the first element of the weight index; Calculate the number of zero elements between non-zero values ​​and adjacent non-zero values ​​in the weights; The number of zero elements between non-zero values ​​and adjacent non-zero values ​​in the weights are used as weight indices for the other elements except the first element.

3. A convolutional neural network accelerator according to claim 2, characterized in that, The sparse weights store non-zero weights in a sequential order.

4. A convolutional neural network accelerator according to claim 1, characterized in that, The input feature index is generated in the following way: Calculate the number of non-zero elements in the input features; The number of non-zero elements in the input features is used as the first element of the input feature index. Calculate the number of zero elements between non-zero values ​​and adjacent non-zero values ​​in the input features; The number of zero elements between non-zero values ​​and adjacent non-zero values ​​in the input features are used as the input feature indices for the other elements except the first element.

5. A convolutional neural network accelerator according to claim 4, characterized in that, The sparse input features store non-zero input feature elements. This accelerator is designed based on the above-mentioned convolutional neural network acceleration method. This acceleration method performs symmetrical processing on the convolution kernel, which is equivalent to compressing the convolution kernel parameters by half. This reduces the space required for weight storage and the number of accesses to external memory. However, after compression, if you want to perform operations through the subsequent multiplier array, you must obtain the weights and input feature indices through sparse encoding. This accelerator connects the entire neural network acceleration process through the encoding unit, so that the acceleration of the convolutional neural network has the characteristics of high computational efficiency and low storage space requirement.

6. A method for accelerating a convolutional neural network based on the convolutional neural network accelerator of claim 1, characterized in that, include: S100. Obtain the model parameters of the pre-trained convolutional neural network, wherein the model parameters include the kernel parameters in each convolutional layer; S200. Calculate the average value of the central symmetric elements of the convolution kernel in each convolutional layer based on the kernel parameters in each convolutional layer, and replace it with the kernel parameters in each convolutional layer; S300. Given an input / output sample pair, train the convolutional neural network based on the input / output sample pair, and end the training process when the specified error range or the maximum number of iterations is reached.

7. The method for accelerating a convolutional neural network according to claim 6, characterized in that, S300 includes the following steps: S310. Given an input / output sample pair; S320. Determine whether the convolutional neural network has reached the maximum number of training iterations. If yes, end the training process; otherwise, execute S330. S330. Use the input sample pairs as input vectors to propagate the convolutional neural network forward and calculate the output of the hidden layer units one by one; S340. Use the output of the hidden layer unit obtained in S330 as the input vector to calculate the output of the output unit; S350. Calculate the error of the output unit; S360. Determine whether the error of the output unit meets the preset condition. If yes, end the training process; otherwise, execute S370. S370. Calculate the error of the hidden layer elements; S380. Determine whether the hidden layer unit of the current calculation error is a convolutional layer. If yes, calculate the center-symmetric error gradient; otherwise, calculate the error gradient. S390. Adjust the weights of the convolution kernel according to the gradient calculated in S380, and then execute S320.

8. The method for accelerating a convolutional neural network according to claim 7, characterized in that, The error of the output unit is the difference between the output of the output unit and the given output sample pair.

9. A method for accelerating convolutional neural networks according to claim 7, characterized in that, The error of the hidden layer element is calculated, including: Based on the error of the output unit, the error of the hidden layer unit is calculated according to the chain rule.

Images