Block-sparse methods, devices, and processing units based on convolutional neural networks

Abstract

This application relates to the field of data processing technology, and provides a block sparse method, apparatus, and processing unit based on convolutional neural networks. The block sparse method based on convolutional neural networks converts the original weights of dimension N×C×K×K into K... 2 The weights are rearranged into C×K values, then divided into multiple data blocks according to a preset dimension. Finally, redundant non-zero values ​​within these data blocks are pruned according to a pre-set data sparsity. For each data block, only data with minimal impact is pruned, reducing the influence of sparsity on training accuracy and achieving high sparsity. Furthermore, the hardware design addresses the imbalance of computational tasks between different processing units in traditional training accelerators when handling sparse tensors, and avoids read / write conflicts in weight data storage during forward and backward propagation.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a block sparsity method, apparatus and processing unit based on convolutional neural networks. Background Technology

[0002] Convolutional Neural Networks (CNNs) have been widely applied in various fields of artificial intelligence in recent years, demonstrating superior performance in image processing. However, training a CNN often requires a large number of multiplication and accumulation operations, which consume significant computational resources and time. When CNNs are deployed on edge devices and further trained using users' private datasets, these numerous multiplication and accumulation operations pose a significant challenge to network deployment.

[0003] One effective solution to this problem is pruning, which removes unimportant weights from the weighted network to reduce the number of parameters. Existing pruning techniques can be divided into two categories: structured pruning and unstructured pruning. Structured pruning is performed on a channel or layer-by-layer basis, resulting in a more regular sparse model that is easy to deploy on traditional hardware for computation, but it can significantly impact the accuracy of the network. Unstructured pruning, on the other hand, can produce a sparse model with better accuracy.

[0004] For unstructured pruning methods, on the one hand, a typical convolutional neural network usually includes a large number of network layers. Among them, the most important network layer is the convolutional layer, which is also the most computationally intensive network layer in the convolutional neural network. Each convolutional layer has a set of weight matrices and a set of input activation matrices. The weight matrix contains a large amount of weight data, and the activation matrix contains a large amount of activation data. For example... Figure 1 As shown, during neural network training, the weight tensors of convolutional layers require transposition during forward and backward propagation. This transposition results in different memory arrangements for storing weight data during forward and backward propagation. In other words, if the weight tensors are arranged according to the data arrangement required for forward propagation before training, they must be rearranged and re-stored in memory during backward propagation to avoid read / write conflicts. Since forward and backward propagation are performed alternately during training, this rearrangement of weight data incurs additional costs and resource consumption.

[0005] On the other hand, for a traditional training accelerator, when dealing with sparse tensors, especially unstructured sparseness, the computational tasks at different processing stages become unbalanced. This is because the position of sparse zero values ​​in the weight tensor is uncertain for unstructured sparseness. This results in different processing units (PEs) needing to perform different numbers of multiply-accumulate operations when processing a block of data. If the accelerator skips all zero-value calculations, it leads to an imbalance in workload between different processing units. If the sparsity of a data block is high, there are more sparse zero values, resulting in fewer effective multiply-accumulate operations and fewer clock cycles required to process that block of data. However, if the sparsity of a data block is low, there are fewer sparse zero values, resulting in more effective multiply-accumulate operations. This imbalance in workload causes data flow disorder between different processing units, preventing the accelerator from completing the computational tasks correctly. However, if the accelerator does not skip the calculation of sparse zero values ​​during computation, the acceleration effect brought by unstructured sparseness will disappear. Summary of the Invention

[0006] To address the problems in existing technologies where, when convolutional neural networks are deployed on edge devices and further trained using users' private datasets, conventional pruning methods cause additional costs and resource consumption due to the rearrangement of weight data during training, or lead to chaotic data flow between different processing units due to increased workload, this application provides a block sparsity method, apparatus, and processing unit based on convolutional neural networks.

[0007] The first aspect of this application provides a block sparsity method based on convolutional neural networks, the block sparsity method based on convolutional neural networks including:

[0008] The original weights of dimension N×C×K×K are converted to K 2 A C×K rearrangement weight, where N is the number of convolutional kernels, C is the number of feature maps per kernel, and K×K is the tensor of a single feature map; using This represents the data points in the original weights, where 0≤i0≤N-1, 0≤i1≤C-1, 0≤i2, i3≤K-1, and uses... This represents the data points whose weights have been rearranged, where 0 ≤ K ≤ K. 2 -1, 0≤r≤C-1, 0≤s≤N-1, M is,

[0009] Based on the preset dimensions, the rearranged weights are divided into multiple data blocks;

[0010] Based on the pre-set data sparsity, redundant non-zero values ​​in the multiple data blocks are removed to obtain compressed weight data, and the sparse mask information of the data blocks is recorded. The sparse mask information includes the index information of the non-zero value weights in the data blocks.

[0011] In one implementation, the step of dividing the rearranged weights into multiple data blocks according to a preset dimension further includes:

[0012] Based on a preset dimension, it is determined whether the rearranged weights can be divided into an integer number of data blocks. If the rearranged weights cannot be divided into an integer number of data blocks, the tensor of the rearranged weights is padded with zeros until the rearranged weights after zero padding can be divided into an integer number of data blocks.

[0013] A second aspect of this application provides a processing unit, characterized in that a plurality of the processing units are arranged in a pulse array, the processing unit comprising:

[0014] The control module is configured to receive sparse mask information, forward propagation information and backward propagation information of the data block. The sparse mask information includes the index information of non-zero weights in the data block.

[0015] It is also configured to generate operation control instructions based on forward propagation information and / or backward propagation information, and to send the operation control instructions and sparse mask information to the multiply-accumulate module;

[0016] The activation data preprocessing module is configured to receive activation data and sparse mask information of data blocks, and to filter out valid activation data from the activation data according to the sparse mask information, and to send the valid activation data to the multiply-accumulate module.

[0017] The multiply-accumulate module is configured to receive compressed weight data from an external source, receive valid activation data from an activation data preprocessing module, and receive corresponding register control signals from a control module.

[0018] It is also configured to perform multiplication and accumulation operations on mutually matched weighted data and activation data to generate partial sums and results;

[0019] The result selection module is configured to receive the part and result from the multiply-accumulate module, and to receive a set of completed convolution operation results from the previous neighboring processing unit.

[0020] It is also configured to receive a result selection signal from the outside and output the part and result or convolution operation result to the next adjacent processing unit according to the result selection signal;

[0021] The compressed weight data is obtained using the following method:

[0022] The original weights of dimension N×C×K×K are converted to K 2 A C×K rearrangement weight, where N is the number of convolutional kernels, C is the number of feature maps per kernel, and K×K is the tensor of a single feature map; using This represents the data points in the original weights, where 0≤i0≤N-1, 0≤i1≤C-1, 0≤i2, i3≤K-1, and uses... This represents the data points whose weights have been rearranged, where 0 ≤ K ≤ K. 2 -1, 0≤r≤C-1, 0≤s≤N-1, M is a flag bit.

[0023] Based on the preset dimensions, the rearranged weights are divided into multiple data blocks;

[0024] Based on the pre-set data sparsity, redundant non-zero values ​​in the multiple data blocks are removed to obtain compressed weight data, and the sparse mask information of the data blocks is recorded.

[0025] In one implementation, the step of dividing the rearranged weights into multiple data blocks according to a preset dimension further includes:

[0026] Based on a preset dimension, it is determined whether the rearranged weight can be divided into an integer number of data blocks. If the rearranged weight cannot be divided into an integer number of data blocks, the tensor of the rearranged weight is padded with zeros until the rearranged weight after zero padding can be divided into an integer number of data blocks.

[0027] In one implementation, the multiply-accumulate module includes: a multiplier, an adder, a first data selection module, a second data selection module, and S memories, wherein the dimension of the data block is S×T, and S≥T;

[0028] The multiplier is configured to receive matching weight values ​​and activation values, and to perform multiplication and accumulation operations on the matching weight data and activation data.

[0029] The adder is configured to add the result of the multiplier operation to a portion of the output of the second data selector and the result;

[0030] The first data selector is configured to store the multiply-accumulate result into the corresponding memory;

[0031] The second data selector is configured to select the appropriate memory and output the portion stored in the appropriate memory and the result to the adder.

[0032] A third aspect of this application provides a processing apparatus, characterized in that the processing apparatus includes a plurality of processing units, the plurality of processing units being arranged in an array to form an arithmetic array, and the processing apparatus further includes:

[0033] The first buffer is configured to store activation data and broadcast the activation data to the processing unit in a line broadcast manner;

[0034] The second buffer is configured to store the compressed weight tensor and sparse mask information, and broadcast the compressed weight tensor and sparse mask information to the processing unit in a column broadcast manner.

[0035] Data routing is configured to control the first and second caches.

[0036] A fourth aspect of this application provides a block sparse device based on a convolutional neural network, characterized in that the block sparse device based on the convolutional neural network includes:

[0037] The rearrangement module is configured to convert the original weights of dimension N×C×K×K into K. 2 A C×K rearrangement weight, where N is the number of convolutional kernels, C is the number of feature maps per kernel, and K×K is the tensor of a single feature map; using This represents the data points in the original weights, where 0≤i0≤N-1, 0≤i1≤C-1, 0≤i2, i3≤K-1, and uses... This represents the data points whose weights have been rearranged, where 0 ≤ K ≤ K. 2 -1, 0≤r≤C-1, 0≤s≤N-1, M is a flag bit.

[0038] The tensor block module is configured to divide the rearranged weights into multiple data blocks according to a preset dimension;

[0039] The sparse module is configured to remove redundant non-zero values ​​in the multiple data blocks according to a pre-set data sparsity, obtain compressed weight data, and record the sparse mask information of the data blocks, wherein the sparse mask information includes the index information of the non-zero value weights in the data blocks.

[0040] One implementation also includes:

[0041] The zero-padding module is configured to determine whether the rearranged weights can be divided into an integer number of data blocks based on a preset dimension. If the rearranged weights cannot be divided into an integer number of data blocks, the tensor of the rearranged weights is zero-padding until the zero-padding rearranged weights can be divided into an integer number of data blocks.

[0042] A fifth aspect of this application provides a computer device, including:

[0043] Memory, used to store computer programs;

[0044] A processor is configured to implement the block sparse method based on a convolutional neural network provided in the first aspect of the embodiments of this application when executing the computer program.

[0045] A sixth aspect of this application provides a computer-readable storage medium storing a computer program, which, when processed and executed, implements the steps of the block sparse method based on a convolutional neural network provided in the first aspect of this application.

[0046] This application provides a block sparse method, apparatus, and processing unit based on convolutional neural networks. The block sparse method based on convolutional neural networks converts the original weights of dimension N×C×K×K into K... 2 The C×K rearranged weights are then divided into multiple data blocks according to a preset dimension. Finally, redundant non-zero values ​​within these data blocks are pruned according to a pre-set data sparsity to obtain compressed weight data. For a single data block, only data with minimal impact on the block is pruned. This pruning method results in random sparsity, which reduces the impact of sparsity on training accuracy, thus achieving high sparsity. Furthermore, the block sparsity method based on the proposed convolutional neural network provided in this application ensures that, during hardware design, when a processing unit processes a multiplication-accumulation operation within an entire data block, the time spent processing each data block is the same because the number of non-zero values ​​within each block is identical. This solves the problem of unbalanced computational tasks between different processing units in traditional training accelerators when processing sparse tensors. Moreover, the same data format of weight data can be used during forward and backward propagation. Therefore, as long as all rearranged weights are stored, no additional transposition is required during computation, and no storage read / write conflicts occur. Attached Figure Description

[0047] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This diagram illustrates the transpose operation required for the weight tensors of convolutional layers during forward and backward propagation when training a neural network.

[0049] Figure 2A flowchart illustrating the block sparse method based on convolutional neural networks provided in this application embodiment;

[0050] Figure 3 This is a schematic diagram of weight tensor rearrangement provided in an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of sparse pruning provided for an embodiment of this application;

[0052] Figure 5 This is a schematic diagram of the structure of a processing unit provided in an embodiment of this application;

[0053] Figure 6 This is a schematic diagram of the structure of the multiply-accumulate module provided in an embodiment of this application;

[0054] Figure 7 A schematic diagram illustrating how the processing unit provided in this application embodiment skips zero-value pair data block operations;

[0055] Figure 8 This is a schematic diagram of a processing device provided in an embodiment of this application. Detailed Implementation

[0056] The first aspect of this application provides a block sparsity method based on convolutional neural networks, such as... Figure 2 As shown, the block sparse method based on convolutional neural networks includes steps 210 to 230.

[0057] Step 210: Convert the original weights of dimension C×K×K into K. 2 A C×K rearrangement weight, where N is the number of convolutional kernels, C is the number of feature maps per kernel, and K×K is the tensor of a single feature map; using This represents the data points in the original weights, where 0≤i0≤N-1, 0≤i1≤C-1, 0≤i2, i3≤K-1, and uses... This represents the data points whose weights have been rearranged, where 0 ≤ K ≤ K. 2 -1, 0≤r≤C-1, 0≤s≤N-1, M is a flag bit used to distinguish it from the previous W, and has no numerical meaning.

[0058] like Figure 3 As shown, taking a convolutional layer in a CNN network as an example, assuming the original weights of the convolutional layer consist of N convolutional kernels, each kernel has C feature maps, and each feature map is a K×K tensor, then the dimension of the original weights for this convolutional layer is N×C×K×K. For the original N C×N×N convolutional kernels, the goal is to map the kernels to K... 2 A C×N tensor. For a data point in the original weights, use... This represents the weights, where 0 ≤ i0 ≤ N-1, 0 ≤ i1 ≤ C-1, 0 ≤ i2, i3 ≤ K-1. For data points in the rearranged weights, use... It means that 0≤k≤K 2 -1, 0≤r≤C-1, 0≤s≤N-1, where M is a flag. Next, the original weights are rearranged, with the coordinates of the data points mapped as follows:

[0059] i0 = s;

[0060] i1 = r;

[0061]

[0062]

[0063] After completing the above operations and mappings, it is possible to:

[0064]

[0065] For the rearranged K 2 There are N weights, and each rearranged weight contains N columns of data, with C data points in each column, each coming from one channel of the N original weights. Therefore, for each rearranged weight, its dimension is C×N.

[0066] Step 220: Divide the rearranged weights into multiple data blocks according to the preset dimensions.

[0067] The step of dividing the rearranged weights into multiple data blocks further includes determining whether the rearranged weights can be divided into an integer number of data blocks based on a preset dimension. If the rearranged weights cannot be divided into an integer number of data blocks, the tensor of the rearranged weights is padded with zeros until the zero-padded rearranged weights can be divided into an integer number of data blocks.

[0068] After rearranging the original weights, it is necessary to... 2 Sparsity is achieved by weighting the rearranged elements. For example... Figure 4As shown, the first step in sparsity optimization is to divide the data into blocks for each rearranged weight. The size of each data block is not fixed and can be chosen according to the actual situation. In this embodiment, the dimension of the data block is generally a power of 2, such as 2×2, 2×4, or 4×4 data blocks. In some cases, if the rearranged weight cannot be completely divided into an integer number of data blocks, the rearranged weight is padded with zeros until it can be divided into an exact integer number of data blocks. Taking a 4×4 data block as an example, if the dimension of the rearranged weight is 4a, it can be divided into blocks. If a dimension of the rearranged weight is 4a+1, 4a+2, or 4a+3, it cannot be completely divided into blocks, and the dimension of the rearranged weight needs to be padded to 4a+4. All the padded data is 0, so it will not affect the original calculation result. Figure 3 The example demonstrates how to divide an 8×12 rearranged weight into 4×4 data blocks. Since both dimensions are multiples of 4, the entire rearranged weight can be divided into 6 data blocks.

[0069] Step 230: According to the preset data sparsity, redundant non-zero values ​​in the multiple data blocks are removed to obtain compressed weight data, and the sparse mask information of the data blocks is recorded. The sparse mask information includes the index information of the non-zero value weights in the data blocks.

[0070] After reordering the weights into blocks, pruning is required to remove redundant non-zero values ​​from the reordered weights. Sparse exploration is performed on a block-by-block basis; for each block, redundant non-zero values ​​are set to zero. Generally, smaller data points within a block can be pruned; this pruning method has minimal impact on accuracy. Within a block, information about the data position can be recorded using various methods. In this embodiment, a coordinate list (COO) is used to represent the position information of non-zero values ​​within a block. Other data representation methods can also be used, such as CSR (Compressed Sparse Column), CSC (Compressed Sparse Row), Bitmap, etc. Figure 7 As shown, for a 4×4 data block, the coordinate list method sequentially defines 16 positions from 0 to 15. Therefore, when representing the coordinates of non-zero values, only a 4-bit width is needed to represent the absolute position of a non-zero value within a data block. Because this method does not need to record the position of a non-zero value in the entire rearranged weights, a smaller bit width can be used to represent the data position, reducing storage costs. On the other hand, because a very direct method is used to represent the data position, when retrieving non-zero values ​​in hardware, there is no need to design an overly complex sparsity detection module, thereby reducing hardware design costs.

[0071] This application provides a block sparsity method based on convolutional neural networks. After rearranging the weights to achieve sparsity, the sparsity within all data blocks is the same, ensuring that each data block contains the same number of non-zero values. For example... Figure 4 As shown, for a 4×4 data block, if half of the redundant values ​​are pruned, after sparsity is achieved, all data blocks will contain 8 non-zero values ​​and 8 zero values. The sparsity for a single data block is 50%, and the sparsity for the entire rearrangement weights is also 50%, as is the sparsity for the weights at this level. For data blocks that were padded with zeros for complete partitioning, the added zero values ​​will also participate in pruning during sparsity. For example, if the original rearrangement weights are 4a+1, some data blocks will contain only 4 non-zero values ​​and 12 zero values ​​before sparsity. If the required sparsity is 50%, the goal is to prune 8 redundant values, but these 4 non-zero values ​​remain unchanged. Therefore, this will have a slight impact on the final sparsity, but this impact is negligible. Figure 4 As shown, for a data block, the data that has a small impact on the data block is pruned. The sparsity resulting from this pruning method is random, which can reduce the impact of sparsity on training accuracy, thus achieving a high sparsity. Furthermore, the block sparsity method based on the proposed convolutional neural network provided in this application, during hardware design, ensures that when a processing unit processes a multiplication-accumulation operation within an entire data block, the number of non-zero values ​​within each data block is the same, therefore the time spent by each processing unit processing one data block is also the same. This solves the problem of unbalanced computational tasks between different processing units in traditional training accelerators when processing sparse tensors.

[0072] The block sparse method based on convolutional neural networks provided in this application also solves the problem of weight transposition during forward and backward propagation in sparse networks. During training, after forward propagation, the weights need to be transposed before backward propagation. Since forward and backward propagation are performed repeatedly during training, this difference in data storage methods can lead to read / write conflicts. Storing both types of data in advance would significantly increase storage costs. The block sparse method based on convolutional neural networks proposed in this application solves the problem caused by weight transposition. For the rearranged weights, all K... 2Data in the same column of each weight represents a convolution kernel. Therefore, during forward propagation, we only need to multiply and accumulate all non-zero values ​​in the same column of the rearranged weights with the corresponding activation data, and then sum the partial sums of the same columns in all rearranged weights to obtain the corresponding convolution result. Similarly, in backpropagation, all K... 2 Data in the same row of weights represents a convolution kernel. Therefore, by multiplying and accumulating all non-zero values ​​in the same row of rearranged weights with the corresponding activation data, the convolution result during backpropagation can be obtained. The advantage of this operation method is that weight data in the same data format can be used for forward and backward propagation. Therefore, as long as all rearranged weights are stored, no additional transposition is required during operation, and no read / write conflicts will occur in storage.

[0073] A second aspect of this application provides a processing unit suitable for processing weight data compressed using the aforementioned block-sparse method based on convolutional neural networks. This processing unit can process operations on a single data block at a time, resolving the imbalance in computational tasks that can occur when training accelerators handle sparse tensors. Furthermore, the processing unit array within the overall hardware structure employs a pulsating array design, which improves data reuse, increases accelerator throughput, and accelerates training speed.

[0074] In the overall hardware design of the training accelerator, the most important aspect is the design of the basic processing unit. In sparse algorithms, data is divided into blocks based on rearranged weights. Assuming the dimension of the data block is A×B, because the forward propagation requires accumulation on a column-by-column basis and the backpropagation requires accumulation on a row-by-row basis, for an A×B data block, the forward propagation will produce B independent parts and results, and the backpropagation will produce A independent parts and results. Therefore, the basic processing unit needs to be able to process these A (or B) independent parts and results simultaneously.

[0075] like Figure 5 As shown, the processing unit includes: a control module, an activation data preprocessing module, a multiply-accumulate module, and a result selection module.

[0076] The control module is configured to receive sparse mask information, forward propagation information, and backward propagation information of a data block, wherein the sparse mask information includes index information of non-zero weights in the data block; it is also configured to generate operation control instructions based on the forward propagation information and / or backward propagation information, and send the operation control instructions and sparse mask information to the multiply-accumulate module.

[0077] The activation data preprocessing module is configured to receive activation data and sparse mask information of data blocks, filter out valid activation data from the activation data according to the sparse mask information, and send the valid activation data to the multiply-accumulate module.

[0078] The multiply-accumulate module is configured to receive compressed weight data from an external source, receive valid activation data from an activation data preprocessing module, receive corresponding register control signals from a control module, and is also configured to perform multiply-accumulate operations on the matched weight data and activation data to generate partial sums and results.

[0079] The result selection module is configured to receive the partial sum and result from the multiply-accumulate module, and to receive a set of completed convolution operation results from the previous adjacent processing unit. It is also configured to receive a result selection signal from the outside and output the partial sum and result or the convolution operation result to the next adjacent processing unit according to the result selection signal.

[0080] Among them, such as Figure 6 As shown, the multiply-accumulate module includes: a multiplier, an adder, a first data selection module, a second data selection module, and S memories, wherein the dimension of the data block is S×T, and S≥T.

[0081] The multiplier is configured to receive mutually matched weight values ​​and activation values, and to perform multiplication and accumulation operations on the mutually matched weight data and activation data.

[0082] The adder is configured to add the result of the multiplier operation to a portion of the output of the second data selector and the result.

[0083] The first data selector is configured to store the multiply-accumulate result into the corresponding memory.

[0084] The second data selector is configured to select a corresponding memory and output the portion stored in the corresponding memory and the result to the adder.

[0085] A third aspect of this application provides a processing apparatus, which includes a plurality of processing units arranged in an array as a computational array. In this embodiment, for example... Figure 8 As shown, all processing units of the processing device are arranged in a 32×32 arithmetic array in the form of a pulse array. In the figure, PE represents a processing unit.

[0086] The processing device further includes:

[0087] The first buffer is configured to store activation data and broadcast the activation data to the processing unit in a line broadcast manner.

[0088] The second buffer is configured to store the compressed weight tensor and sparse mask information, and broadcast the compressed weight tensor and sparse mask information to the processing unit in a column broadcast manner.

[0089] Data routing is configured to control the first and second buffers. A data router will control the two buffers to correctly output the results to the computation array.

[0090] In this embodiment, row broadcasting refers to broadcasting on a row-by-row basis of the computation array, and column broadcasting refers to broadcasting on a column-by-column basis of the computation array.

[0091] A fourth aspect of this application provides a block sparse device based on a convolutional neural network, the block sparse device based on the convolutional neural network comprising:

[0092] The rearrangement module is configured to convert the original weights of dimension N×C×K×K into K. 2 A C×K rearrangement weight, where N is the number of convolutional kernels, C is the number of feature maps per kernel, and K×K is the tensor of a single feature map; using This represents the data points in the original weights, where 0≤i0≤N-1, 0≤i1≤C-1, 0≤i2, i3≤K-1, and uses... This represents the data points whose weights have been rearranged, where 0 ≤ K ≤ K. 2 -1, 0≤r≤C-1, 0≤s≤N-1, M is a flag bit.

[0093] The tensor block module is configured to divide the rearranged weights into multiple data blocks according to a preset dimension.

[0094] The sparse module is configured to remove redundant non-zero values ​​in the multiple data blocks according to a pre-set data sparsity, obtain compressed weight data, and record the sparse mask information of the data blocks, wherein the sparse mask information includes the index information of the non-zero value weights in the data blocks.

[0095] The zero-padding module is configured to determine whether the rearranged weights can be divided into an integer number of data blocks based on a preset dimension. If the rearranged weights cannot be divided into an integer number of data blocks, the tensor of the rearranged weights is zero-padding until the zero-padding rearranged weights can be divided into an integer number of data blocks.

[0096] A fifth aspect of this application provides a computer device, including:

[0097] Memory is used to store computer programs.

[0098] A processor is configured to execute the computer program to implement the block sparse method based on convolutional neural networks provided in the first aspect of this application.

[0099] A sixth aspect of this application provides a computer-readable storage medium storing a computer program, which, when processed and executed, implements the steps of the block sparse method based on a convolutional neural network provided in the first aspect of this application.

[0100] This application provides a block sparse method, apparatus, and processing unit based on convolutional neural networks. The block sparse method based on convolutional neural networks converts the original weights of dimension N×C×K×K into K... 2 The C×K rearranged weights are then divided into multiple data blocks according to a preset dimension. Finally, redundant non-zero values ​​within these data blocks are pruned according to a pre-set data sparsity to obtain compressed weight data. For a single data block, only data with minimal impact on the block is pruned. This pruning method results in random sparsity, which reduces the impact of sparsity on training accuracy, thus achieving high sparsity. Furthermore, the block sparsity method based on the proposed convolutional neural network provided in this application ensures that, during hardware design, when a processing unit processes a multiplication-accumulation operation within an entire data block, the time spent processing each data block is the same because the number of non-zero values ​​within each block is identical. This solves the problem of unbalanced computational tasks between different processing units in traditional training accelerators when processing sparse tensors. Moreover, the same data format of weight data can be used during forward and backward propagation. Therefore, as long as all rearranged weights are stored, no additional transposition is required during computation, and no storage read / write conflicts occur.

[0101] The above specific embodiments further illustrate the purpose, technical solution and beneficial effects of this application. It should be understood that the above are only specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

Claims

1. A processing unit, characterized in that, Multiple processing units are arranged in a pulsating array, and each processing unit includes: The control module is configured to receive sparse mask information, forward propagation information and backward propagation information of the data block. The sparse mask information includes the index information of non-zero weights in the data block. It is also configured to generate operation control instructions based on forward propagation information and / or backward propagation information, and to send the operation control instructions and sparse mask information to the multiply-accumulate module; The activation data preprocessing module is configured to receive activation data and sparse mask information of data blocks, and skip zero-value activation data; and based on the sparse mask information, filter out valid activation data from the activation data, and send the valid activation data to the multiply-accumulate module. The multiply-accumulate module includes: a multiplier, an adder, a first data selection module, a second data selection module, and S memory locations, wherein the dimension of the data block is... ,and ; The multiplier is configured to receive matching weight values ​​and activation values, and to perform multiplication and accumulation operations on the matching weight data and activation data. The adder is configured to add the result of the multiplier operation to a portion of the output of the second data selector and the result; The first data selector is configured to store the multiply-accumulate result into the corresponding memory; The second data selector is configured to select the corresponding memory and output the portion stored in the corresponding memory and the result to the adder; S memories are configured to store portions and results from different channels; The result selection module is configured to receive the part and result from the multiply-accumulate module, and to receive a set of completed convolution operation results from the previous neighboring processing unit. It is also configured to receive a result selection signal from the outside and output the part and result or convolution operation result to the next adjacent processing unit according to the result selection signal; The compressed weight data is generated through the following hardware adaptation method: Dimension is The original weights are converted to indivual The rearrangement weights, where, The number of convolution kernels, The number of feature maps for a single convolutional kernel. Tensors of a single feature map; using Represents the data points in the original weights, where -1, -1, , -1, using This represents the data points whose weights have been rearranged, where... -1, -1, -1, For flag position, ; Based on the preset dimensions, the rearranged weights are divided into multiple data blocks; Based on the pre-set data sparsity, redundant non-zero values ​​in the multiple data blocks are removed to obtain compressed weight data, and the sparse mask information of the data blocks is recorded.

2. The processing unit according to claim 1, characterized in that, The step of dividing the rearranged weights into multiple data blocks according to a preset dimension further includes: Based on a preset dimension, it is determined whether the rearranged weights can be divided into an integer number of data blocks. If the rearranged weights cannot be divided into an integer number of data blocks, the tensor of the rearranged weights is padded with zeros until the rearranged weights after zero padding can be divided into an integer number of data blocks.

3. The processing unit according to claim 1, characterized in that, The multiply-accumulate module includes: a multiplier, an adder, a first data selection module, a second data selection module, and S memory locations, wherein the dimension of the data block is... ,and ; The multiplier is configured to receive matching weight values ​​and activation values, and to perform multiplication and accumulation operations on the matching weight data and activation data. The adder is configured to add the result of the multiplier operation to a portion of the output of the second data selector and the result; The first data selector is configured to store the multiply-accumulate result into the corresponding memory; The second data selector is configured to select the appropriate memory and output the portion stored in the appropriate memory and the result to the adder.

4. A processing apparatus, characterized in that, The processing apparatus includes a plurality of processing units as described in any one of claims 1 to 3, wherein the plurality of processing units are arranged in an array as a computation array, and the processing apparatus further includes: The first buffer is configured to store activation data and broadcast the activation data to the processing unit in a line broadcast manner; The second buffer is configured to store the compressed weight tensor and sparse mask information, and broadcast the compressed weight tensor and sparse mask information to the processing unit via column broadcast. Data routing is configured to control the first and second caches.

5. A block sparsity method based on convolutional neural networks, characterized in that, The block sparsity method based on convolutional neural networks is implemented by the processing unit according to any one of claims 1-3, and the block sparsity method based on convolutional neural networks includes: Dimension is The original weights are converted to indivual The weights are rearranged to ensure that forward and backward propagation share the same data format, where, The number of convolution kernels, The number of feature maps for a single convolutional kernel. Tensors of a single feature map; using Represents the data points in the original weights, where -1, -1, , -1, using This represents the data points whose weights have been rearranged, where... -1, -1, -1, For flag position, ; The preset dimension is determined based on the block processing capability of the processing unit, and the rearranged weights are divided into multiple data blocks according to the preset dimension. Based on the pre-set data sparsity, redundant non-zero values ​​in the multiple data blocks are removed to obtain compressed weight data, and the sparse mask information of the data blocks is recorded. The sparse mask information includes the index information of the non-zero value weights in the data blocks.

6. The block sparsity method based on convolutional neural networks according to claim 5, characterized in that, The step of dividing the rearranged weights into multiple data blocks according to a preset dimension further includes: Based on a preset dimension, it is determined whether the rearranged weights can be divided into an integer number of data blocks. If the rearranged weights cannot be divided into an integer number of data blocks, the tensor of the rearranged weights is padded with zeros until the rearranged weights after zero padding can be divided into an integer number of data blocks.

7. A block sparse device based on a convolutional neural network, characterized in that, The convolutional neural network-based block sparsity device is used to implement the convolutional neural network-based block sparsity method according to any one of claims 5-6, wherein the convolutional neural network-based block sparsity device comprises: The rearrangement circuit is configured to receive the original weight tensor and rearrange it to dimension 1. The original weights are converted to indivual The rearranged weights are used to output rearranged weights with a uniform forward and / or backward propagation format, where, The number of convolution kernels, The number of feature maps for a single convolutional kernel. Tensors of a single feature map; using Represents the data points in the original weights, where -1, -1, , -1, using This represents the data points whose weights have been rearranged, where... -1, -1, -1, For flag position, ; Tensor block circuits are configured to divide rearranged weights into multiple data blocks according to a preset dimension; The sparse circuit is configured to remove redundant non-zero values ​​in the plurality of data blocks according to a preset data sparsity, obtain compressed weight data, and record the sparse mask information of the data blocks, wherein the sparse mask information includes the index information of the non-zero value weights in the data blocks.

8. The block sparse device based on a convolutional neural network according to claim 7, characterized in that, Also includes: The zero-padding module is configured to determine whether the rearranged weights can be divided into an integer number of data blocks based on a preset dimension. If the rearranged weights cannot be divided into an integer number of data blocks, the tensor of the rearranged weights is zero-padding until the zero-padding rearranged weights can be divided into an integer number of data blocks.

9. A computer device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the block sparse method based on a convolutional neural network as described in any one of claims 5-6 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when processed and executed, implements the steps of the block sparse method based on a convolutional neural network as described in any one of claims 5-6.

Images