A convolutional neural network acceleration system based on unstructured sparse
By designing a convolutional neural network acceleration system based on unstructured sparseness, using flexible on-chip network and configurable accumulation modules, the efficiency problem of unstructured sparse neural network on the accelerator is solved, and efficient convolutional calculation and Optimization of hardware resources.
Patent Information
- Application Number
- CN202110460210.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-27
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-04-27
AI Technical Summary
Existing accelerators have difficulty in effectively accelerating unstructured sparse neural networks, especially because of the irregularity of their non-zero parameter distribution, which makes it impossible to efficiently support sparse computing, resulting in the inability to map the compression ratio of the model to the hardware execution speed.
A convolutional neural network acceleration system based on unstructured sparseness is designed, including off-chip storage, bias buffer area, weight buffer area, activation buffer area, processing module, input overlap area, configurable accumulation module and top-level control The device supports convolutional calculation through flexible on-chip network and configurable accumulation modules, and realizes efficient multiplication and accumulation operations through specific convolutional calculation methods and devices.
It realizes efficient acceleration of unstructured sparse neural networks, reduces the addition tree delay in the calculation process, improves the computing efficiency, and supports convolution and sparseness of any size, and improves hardware utilization.
Smart Images

Figure CN113158132B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of accelerator technology, and in particular to a convolutional neural network acceleration system based on unstructured sparseness. Background Art
[0002] Currently, DCNN (Deep convolutional neural networks) are widely used in many computer vision tasks, such as image classification and object recognition. In order to achieve better performance, DCNN models tend to become wider and deeper, which introduces a large number of increased parameters, resulting in a large increase in computational complexity and storage requirements. Network compression techniques, such as pruning, can significantly reduce the computational complexity of the model while maintaining acceptable accuracy. Pruning algorithms can be divided into two categories: structured sparsity and unstructured sparsity. Compared with structured sparsity, unstructured sparsity can achieve better compression efficiency at the same accuracy, but at the same time there are more irregularities in the distribution of non-zero parameters.
[0003] In specific implementation, due to the inefficient performance or large amount of energy consumption of DCNN, general acceleration platforms such as CPU and GPU cannot achieve the expected acceleration effect. Therefore, many dedicated hardware accelerators based on field-programmable gate array (FPGA) and application-specific integrated circuit (ASIC) platforms have been developed to achieve better energy efficiency and faster processing speed. At present, accelerators are mainly aimed at dense neural networks or structured sparse neural networks with computational rules. For unstructured sparse neural networks, due to the irregular distribution of non-zero data, how to efficiently support sparse computing has become a problem for dedicated accelerators. Without special design, unstructured sparse neural networks cannot map the compression ratio to the hardware execution speed. If too many special designs are made, the hardware resource redundancy is huge and insufficient to achieve efficient acceleration. Therefore, how to achieve effective acceleration for unstructured sparse neural networks is an urgent problem to be solved. Summary of the invention
[0004] The present application provides a convolutional neural network acceleration system based on unstructured sparseness to solve the problem that existing accelerators cannot achieve effective acceleration.
[0005] In a first aspect of the present application, a convolutional neural network acceleration system based on unstructured sparseness is provided, comprising: an off-chip storage and an acceleration device connected to the off-chip storage network, wherein the off-chip storage is used to store convolution calculation data required for convolution calculation, and the convolution calculation data at least includes bias data, weight data, and activation data;
[0006] The acceleration device comprises:
[0007] A bias buffer area, used to obtain the bias data transmitted by the off-chip storage;
[0008] A weight buffer area, used to obtain the weight data transmitted by the off-chip storage;
[0009] An activation buffer area, used to obtain activation data transmitted by the off-chip storage, wherein the activation data corresponds to the weight data;
[0010] A processing module, comprising a plurality of processing units, wherein the processing units are used to obtain the convolution calculation data and perform calculations;
[0011] An input overlap area, arranged between the activation buffer area and the processing module, for storing input overlap data, the input overlap data being used as input for the next calculation;
[0012] A configurable accumulation module, used for accumulating the partial sums output by the processing unit;
[0013] A post-processing module, used for performing linear rectification and pooling processing on the accumulated partial sums, and writing the processing results back to the activation buffer area so as to be used in the next calculation;
[0014] A top-level controller is connected to the bias buffer, the weight buffer and the activation buffer, and is used to control the operation of the bias buffer, the weight buffer and the activation buffer.
[0015] Optionally, the weight cache area includes multiple weight storage bodies, and the output end of the weight storage body is connected to the input end of the processing unit by a cross routing structure, so that the processing unit can obtain weight data from any one of the weight storage bodies.
[0016] Optionally, the activation cache area includes a plurality of activation storage bodies;
[0017] The output end of the activation storage body is connected to the input end of the processing unit in a unicast structure so that the processing unit obtains activation data from the corresponding activation storage body;
[0018] The output end of the processing unit is connected to the input end of the activated storage body in a unicast structure, so that the processing unit writes the processing result back to the corresponding activated storage body.
[0019] Optionally, it also includes: a weight decoder; the weight decoder is arranged in the weight buffer area, and is used to decode the weight data.
[0020] Optionally, the processing unit includes multiple multiplication and accumulation units.
[0021] In a second aspect of the present application, a convolution calculation method is provided, the method being applied to a processing unit provided by any possible implementation of the first aspect, the processing unit comprising a plurality of multiplication and accumulation units, the method comprising:
[0022] Obtain target weight data, where the size of the target weight data is the same as the size of a unit convolution kernel, where the unit convolution kernel is the minimum size of a convolution kernel used for convolution calculation;
[0023] Acquire each target non-zero element in the target weight data and position information of each of the target non-zero elements;
[0024] The target non-zero elements are serially fed into the target multiplication-accumulation units in the order of the position information, and each of the target multiplication-accumulation units is used to calculate the multiplication-accumulation operation of at least one row of elements in the target weight data;
[0025] Selecting target activation data corresponding to the target non-zero element to enter the target multiplication-accumulation unit;
[0026] Performs a multiply-accumulate operation.
[0027] Optionally, the step of performing the multiplication-accumulation operation includes:
[0028] Calculate the product of the current target non-zero element and its corresponding target activation data as the current product;
[0029] Determine whether the difference between the row coordinate value corresponding to the current target non-zero element and the row coordinate value of the previous target non-zero element is zero;
[0030] If so, add the current product to the previous product;
[0031] If not, the moving position of the current product is determined according to the difference between the row coordinates corresponding to the current target non-zero element and the row coordinates of the previous target non-zero element.
[0032] In a third aspect of the present application, a convolution computing device is provided, the device being applied to a processing unit provided in any possible implementation of the first aspect, the processing unit comprising a plurality of multiplication and accumulation units, the device comprising:
[0033] A first acquisition module is used to acquire target weight data, where the size of the target weight data is the same as the size of the unit convolution kernel;
[0034] A second acquisition module is used to acquire each target non-zero element in the target weight data and position information of each target non-zero element;
[0035] An input module, used for serially inputting the target non-zero elements into a target multiplication-accumulation unit according to the sequence of the position information, each of the target multiplication-accumulation units being used for calculating the multiplication-accumulation operation of at least one row of elements in the target weight data;
[0036] A selection module, used for selecting target activation data corresponding to the target non-zero element to enter the target multiplication-accumulation unit;
[0037] The multiply-accumulate module is used to perform multiply-accumulate operations.
[0038] Optionally, the multiplication-accumulation module includes:
[0039] A calculation unit, used for calculating the product of the current target non-zero element and its corresponding target activation data as the current product;
[0040] A judging unit, used to judge whether the difference between the row coordinate value corresponding to the current target non-zero element and the row coordinate value of the previous target non-zero element is zero;
[0041] an accumulating unit, configured to add a current product to a previous product when the judging unit determines that the difference between the row coordinate value corresponding to the current target non-zero element and the row coordinate value of the previous target non-zero element is zero;
[0042] A moving unit is used to determine the moving position of the current product according to the difference between the row coordinates corresponding to the current target non-zero element and the row coordinates of the previous target non-zero element when the judgment unit determines that the difference between the row coordinates corresponding to the current target non-zero element and the row coordinates of the previous target non-zero element is not zero.
[0043] It can be seen from the above technical solutions that the present application provides a convolutional neural network acceleration system based on unstructured sparseness, including: off-chip storage and an acceleration device connected to the off-chip storage network. The acceleration device includes: a bias buffer, a weight buffer, an activation buffer, a processing module, an input overlap area, a configurable accumulation module, a post-processing module and a top-level controller. In the present application, a flexible on-chip network is arranged between the cache area and the processing module, so that the calculation process in this scheme is more flexible. In addition, the configurable accumulation module can reduce the delay of the addition tree in the calculation process and improve the calculation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0045] Figure 1 A schematic diagram of the structure of a convolutional neural network acceleration system based on unstructured sparseness provided in an embodiment of the present application;
[0046] Figure 2 A schematic diagram of a flexible on-chip network structure provided for an embodiment of the present application;
[0047] Figure 3 A schematic diagram of the structure of the accumulation module provided in the embodiment of the present application;
[0048] Figure 4 A flowchart of a convolution calculation method provided in an embodiment of the present application;
[0049] Figure 5 is an example of non-sparse weight data;
[0050] Figure 6 is an example of sparse weight data;
[0051] Figure 7 A structural schematic diagram of a convolution computing device is a structural schematic diagram of a convolution computing device. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0053] Unstructured sparsity for DCNN has been highly anticipated in hardware acceleration due to its large-scale compression of computational complexity. However, due to the high irregularity of the distribution of parameters after the unstructured sparsity process, different problems will be introduced when implementing it on general acceleration platforms (CPU, GPU) and dedicated acceleration platforms for dense neural networks: 1) load imbalance between processing elements (PEs); 2) memory access conflicts for on-chip storage with limited bandwidth. For the acceleration of unstructured sparse DCNN, the above platforms cannot achieve the ideal acceleration effect that matches its reduced computational intensity.
[0054] See also Figure 1As shown in the structural schematic diagram, the embodiment of the present application provides a convolutional neural network acceleration system based on unstructured sparseness, including: off-chip storage and an acceleration device connected to the off-chip storage network. The off-chip storage is used to store convolution calculation data required for convolution calculation, and the convolution calculation data includes at least bias data, weight data, and activation data.
[0055] When processing calculations, configuration information is first read into the top-level controller, which can be used to reconfigure the flexible on-chip network. Activation data and weight data are fetched from the off-chip DRAM to the on-chip SRAM by the memory direct access module. The weight data is the effective weight, which is composed of pruned non-zero weights and zero-value weights (filling the idle cycles caused by unbalanced load). The top-level controller starts the processing after receiving the end signal of the convolution calculation data transmission. The activation data is read in according to the location index of the corresponding weight data, and the weight data and related activation data are input into G_max parallel processing units, where each PE consists of P×P multiplier and accumulator (MAC) units. The design can support convolutions of arbitrary sizes and sparsity.
[0056] In each cycle, the MAC in the PE takes the same weight and inputs from the same input feature plane Ph×(Pw+convolution kernel side length-stride length). After processing, it will produce Ph×Pw partial sums of the same output feature plane, and (convolution kernel side length-stride length)×Pw partial sums for overlapping.
[0057] After the PE is calculated, the partial sum result is input into the configurable accumulator module to complete the accumulation operation on a single output channel. The accumulation module consists of an addition tree and an accumulator array, and can also be used to calculate the shortcut connection addition in the bottleneck-liked operation (BLO) structure. The partial sum result is processed by the linear rectification function (ReLU) and the maximum pooling layer and written back to the activation buffer. After calculating a BLO structure, the calculation result is transferred back to the external large-capacity storage unit.
[0058] According to the above calculation process, the acceleration device is designed, and the acceleration device includes:
[0059] A bias RAM, used to obtain bias data transmitted by the off-chip storage;
[0060] A weight RAM, used to obtain the weight data transmitted by the off-chip storage;
[0061] An activation RAM, used to obtain activation data transmitted from the off-chip storage, wherein the activation data corresponds to the weight data;
[0062] A processing module, comprising a plurality of processing units, wherein the processing units are used to obtain the convolution calculation data and perform calculations;
[0063] An input overlap area (InOV), arranged between the activation buffer area and the processing module, for storing input overlap data, the input overlap data being used as input for the next calculation;
[0064] A configurable accumulation module, used for accumulating the partial sums output by the processing unit;
[0065] A post-processing module, used for performing linear rectification and pooling processing on the accumulated partial sums, and writing the processing results back to the activation buffer area so as to be used in the next calculation;
[0066] A top-level controller is connected to the bias buffer, the weight buffer and the activation buffer, and is used to control the operation of the bias buffer, the weight buffer and the activation buffer.
[0067] In an embodiment of the present application, when the weight data is specially encoded, the acceleration device also includes a weight decoder; the weight decoder is arranged in the weight buffer area, and is used to decode the weight data.
[0068] In the embodiment of the present application, in order to make the computing process more flexible, a flexible on-chip network is designed, see Figure 2 The structural diagram shown in Figure 1 is as follows. The activation buffer and weight buffer are divided into corresponding G_max partitions. The width of each weight buffer is the width of an effective weight, and the width of each activation buffer is an activation block of Ph×Pw. The on-chip network between the buffer and the PE is divided into three parts. The first part is the connection between the G_max weight storage bodies and the G_max processing units through cross routing, which is called the weight input network (weight data feed network, weight DFN); the second part is the connection between the activation buffer and the processing unit, and the data of the G_max activation storage bodies are connected to the G_max processing units in a unicast form; the third part is the output of the G_max processing units connected to the G_max activation storage bodies through a unicast structure.
[0069] In the on-chip network between the cache area and the PE, the processing unit can obtain weight data from any of the weight storage bodies, and can obtain activation data from the corresponding activation storage body. In addition, the processing unit writes the processing results back to the corresponding activation storage body. It can be seen that the flexible on-chip network design makes the calculation process in this solution more flexible.
[0070] Two adjacent PEs can be configured to perform convolution calculations on the same input feature plane, in which case they are clustered into one PE_G. Otherwise, when PEs are divided into different PE_Gs, that is, when processing different feature maps, the corresponding parts and connections of each PE input zero values.
[0071] After calculating the partial sum, the results in different PEs need to be accumulated according to the instructions of the data flow. Since the PEs on the chip can be divided into different PE_Gs, the final accumulator also needs to consider multiple accumulation situations. When each PE is assigned as an independent input channel, the results in different PEs need to be accumulated, and the addition tree after the PE calculation must first ensure this feature. When several PEs are mapped into the same PE_G, the required accumulation process is reduced, and the addition tree with a fixed number of levels increases the overall delay and reduces the hardware calculation efficiency. Therefore, a configurable accumulation module is used in this solution. For the specific structure, see Figure 3 The accumulation module includes a flexible addition tree with selectable levels and an accumulator, wherein the flexible addition tree with selectable technology is used to reduce the overall calculation delay.
[0072] like Figure 3 As shown, it is assumed that there are 8 independent processing units on the chip;
[0073] When the parallelism value is 1, 8 processing units are configured as 1 calculation group to perform convolution calculation on 1 feature map and output 8 calculation results;
[0074] When the parallelism value is 2, 8 processing units are configured as 2 computing groups to perform convolution calculations on 2 feature maps, that is, processing unit 0, processing unit 4, processing unit 2, and processing unit 6 are configured as 1 computing group, and processing unit 1, processing unit 5, processing unit 3, and processing unit 7 are configured as 1 computing group, and each group outputs 4 calculation results;
[0075] When the parallelism value is 4, the 8 processing units are configured as 4 computing groups on average to perform convolution calculations on the 4 feature maps, that is, processing unit 0 and processing unit 4 are configured as 1 computing group, processing unit 1 and processing unit 5 are configured as 1 computing group, processing unit 2 and processing unit 6 are configured as 1 computing group, processing unit 3 and processing unit 7 are configured as 1 computing group, and each group outputs 2 calculation results;
[0076] When the parallelism value is 8, 8 processing units are evenly configured into 8 computing groups to perform convolution operations on 8 feature maps, and each group outputs 1 calculation result.
[0077] The degree of parallelism of the processing unit during runtime is selected according to the number of channels of the input feature map. Figure 3 In the example, saturated truncation 1_0, 1_1, 1_2, and 1_3 are the primary structure of the addition tree, saturated truncation 2_0 and 2_1 are the secondary structure of the addition tree, and saturated truncation 3_0 is the tertiary structure of the addition tree. The selector can select different accumulation methods according to the degree of parallelism, and input the calculation results into the accumulator for accumulation.
[0078] In an embodiment of the present application, the on-chip network provides sufficient on-chip bandwidth to support deep separation convolution and fully connected layers in addition to different forms of convolution. For deep separation convolution, the connection from the buffer area to the processing unit is PE_G that matches the convolution shape, and the configuration from the processing unit to the activation buffer area is PE_G=1. When processing the fully connected layer, compared with the convolution, the weight data is stored in the original activation buffer area, and the activation data is retained in the original weight buffer. The storage location of the buffer area is interchanged, and the corresponding connection network can support the calculation of the fully connected layer without additional hardware consumption.
[0079] When processing sparse weights, the present application embodiment provides a convolution calculation method, see Figure 4 The working flow diagram shown in the figure, the method comprises the following steps:
[0080] Step 101, obtain target weight data, the size of the target weight data is the same as the size of the unit convolution kernel, and the unit convolution kernel is the minimum size of the convolution kernel used for convolution calculation.
[0081] The PE architecture only supports partial and shift convolutions with kernel size less than or equal to 3×3, where the unit kernel size is 3×3. When processing kernels larger than 3×3, a synthesis technique is used to decompose kernels larger than 3×3 into multiple 3×3 sub-kernels by introducing additional zero values in the kernel. When this method is implemented through the architecture in this solution, the additional zero values in the sub-kernels can be treated as sparse weights. Therefore, the proposed architecture can support convolution calculations of arbitrary sizes without causing underutilization of the hardware.
[0082] In this step, it is assumed that the size of the unit convolution kernel is 3×3. If the size of the weight data is larger than 3×3, it is necessary to select target weight data with the same size as the sub-convolution kernel from the weight data according to the size of the unit convolution kernel for convolution calculation.
[0083] Step 102, obtaining each target non-zero element in the target weight data and position information of each of the target non-zero elements.
[0084] Step 103, the target non-zero elements are serially fed into a target multiplication-accumulation unit in the order of the position information, and each target multiplication-accumulation unit is used to calculate the multiplication-accumulation operation of at least one row of elements in the target weight data.
[0085] Step 104, select the target activation data corresponding to the target non-zero element to enter the target multiplication and accumulation unit.
[0086] Step 105, performing a multiplication-accumulation operation.
[0087] In one possible implementation, the steps of performing the multiplication-accumulation operation include:
[0088] Step 1041, calculating the product of the current target non-zero element and its corresponding target activation data as the current product;
[0089] Step 1042, determining whether the difference between the row coordinate value corresponding to the current target non-zero element and the row coordinate value of the previous target non-zero element is zero;
[0090] Step 1043, if the difference between the row coordinate value corresponding to the current target non-zero element and the row coordinate value of the previous target non-zero element is zero, then add the current product to the previous product;
[0091] Step 1044, if the difference between the row coordinate value corresponding to the current target non-zero element and the row coordinate value of the previous target non-zero element is not zero, determine the moving position of the current product according to the difference between the row coordinate value corresponding to the current target non-zero element and the row coordinate value of the previous target non-zero element.
[0092] A 3*3 convolution kernel is used as a specific example for explanation below.
[0093] For non-sparse weights, see Figure 5 In the example of non-sparse weight data shown, W11, W12, W13, W21, W22, W23, W31, W32, W33 are non-sparse weight data, A11, A12, A13, A21, A22, A23, A31, A32, A33 are activation data, and PE1, PE2, PE3 are processing units. The calculation process is as follows:
[0094] (1) The weights are arranged in a row according to their positions in the convolution kernel, in the order of W33, W32, W31, W23, W22, W21, W13, W12, and W11, and enter the MAC unit in series;
[0095] (2) PE3 unit calculates the convolution result of row W3 and row A3. First, weight W33 enters the convolution unit. According to the position of the weight, activation A33 is selected to enter the convolution unit, and A33*W33 is calculated. Then W32 and W31 enter the convolution unit, and the convolution result is: C3 = W31*A31+W32*A32+W33*A33;
[0096] (3) Next, the result C3 of PE3 is vertically transferred to PE2, and the accumulated result C2 = W21*A21+W22*A22+W23*A23 of row W2 and row A2 is accumulated to obtain C2+C3.
[0097] (4) Finally, the result C2+C3 of PE2 is vertically transferred to PE1, and the accumulated result C1=W11*A11+W12*A12+W13*A13 of row W1 and row A1 is accumulated to obtain C1+C2+C3, thus completing the whole convolution calculation process.
[0098] For sparse weights, see Figure 6 In the example of sparse weight data shown, W12, W21, W22, W23, and W33 are all zero values. The calculation process is as follows:
[0099] (1) The weights are arranged in a row according to their positions in the convolution kernel, in the order of W33, W32, W31, W23, W22, W21, W13, W12, and W11, and enter the MAC unit in series;
[0100] (2) PE3 unit calculates the convolution result of row W3 and row A3. First, weight W33 enters the convolution unit. Since W33 is zero, it can be skipped and the calculation of the next element can be directly entered. According to the position of W32, A32 is activated and enters the convolution unit, and A32*W32 is calculated. Then W31 and A31 enter the convolution unit and the convolution result is: C3 = W31*A31 + W32*A32;
[0101] (3) Next, since W21, W22, and W23 are all zero values, the result of the third row PE3 can be directly transferred vertically to PE1, and the accumulated results of rows W1 and A1 are continued to be accumulated C1=W11*A11+W12*A12+W13*A13 to obtain C1+C2+C3, completing the entire convolution calculation process.
[0102] In actual applications, only the non-zero elements in the weight data enter the MAC unit, and the convolution calculation automatically skips the calculation of zero-value elements, thereby improving the calculation efficiency.
[0103] It can be seen from the above calculation process that the convolution calculation provided by this application can support sparse convolution, and a PE unit is designed that can skip zero value calculation to improve the calculation efficiency.
[0104] See also Figure 7 As shown in the structural schematic diagram, the embodiment of the present application provides a convolution computing device, the device comprising:
[0105] An acquisition module 100 is used to acquire target weight data, where the size of the target weight data is the same as the size of the unit convolution kernel;
[0106] An input module 200 is used to serially input each element in the target weight data into a target multiplication-accumulation unit according to its position, and each target multiplication-accumulation unit is used to calculate the multiplication-accumulation operation of at least one row of elements in the target weight data;
[0107] A selection module 300, configured to select target activation data corresponding to the target weight data and enter the target multiplication and accumulation unit;
[0108] The multiplication-accumulation module 400 is used to perform multiplication-accumulation operations.
[0109] Optionally, the multiplication-accumulation module includes:
[0110] A judging unit, used to judge whether a target element used in a current multiplication-accumulation operation is a zero value, wherein the target element is an element in the target weight data used in the current multiplication-accumulation operation;
[0111] The execution unit is used to skip the target element used in the current multiplication and accumulation operation and execute the multiplication and accumulation operation of the next element when the judgment unit determines that the target element used in the current multiplication and accumulation operation is zero.
[0112] The present application is described in detail above in conjunction with specific implementation methods and exemplary examples, but these descriptions cannot be understood as limiting the present application. Those skilled in the art understand that, without departing from the spirit and scope of the present application, a variety of equivalent replacements, modifications or improvements can be made to the technical solution of the present application and its implementation methods, all of which fall within the scope of the present application. The scope of protection of the present application shall be subject to the attached claims.
Claims
1. A convolutional neural network acceleration system based on unstructured sparseness, characterized in that: include: An off-chip storage and an acceleration device connected to the off-chip storage network, wherein the off-chip storage is used to store convolution calculation data required for convolution calculation, and the convolution calculation data at least includes bias data, weight data, and activation data; The acceleration device comprises: A bias buffer area, used to obtain the bias data transmitted by the off-chip storage; A weight buffer area, used to obtain the weight data transmitted by the off-chip storage; An activation buffer area, used to obtain activation data transmitted by the off-chip storage, wherein the activation data corresponds to the weight data; A processing module, comprising a plurality of processing units, wherein the processing units are used to obtain the convolution calculation data and perform calculations; An input overlap area, arranged between the activation buffer area and the processing module, for storing input overlap data, the input overlap data being used as input for the next calculation; A configurable accumulation module, used for accumulating the partial sums output by the processing unit; A post-processing module, used for performing linear rectification and pooling processing on the accumulated partial sums, and writing the processing results back to the activation buffer area so as to be used in the next calculation; A top-level controller is connected to the bias buffer, the weight buffer and the activation buffer, and is used to control the operation of the bias buffer, the weight buffer and the activation buffer.
2. The unstructured sparse convolutional neural network acceleration system according to claim 1, characterized in that: The weight cache area includes multiple weight storage bodies, and the output ends of the weight storage bodies are connected to the input ends of the processing units by a cross-routing structure so that the processing unit can obtain weight data from any one of the weight storage bodies.
3. The unstructured sparse convolutional neural network acceleration system according to claim 1, characterized in that: The activation buffer area includes a plurality of activation storage bodies; The output end of the activation storage body is connected to the input end of the processing unit in a unicast structure so that the processing unit obtains activation data from the corresponding activation storage body; The output end of the processing unit is connected to the input end of the activated storage body in a unicast structure, so that the processing unit writes the processing result back to the corresponding activated storage body.
4. The unstructured sparse convolutional neural network acceleration system according to claim 1, characterized in that: Also includes: Weight decoder; the weight decoder is arranged in the weight buffer area, and is used for decoding the weight data.
5. The unstructured sparse convolutional neural network acceleration system according to claim 1, characterized in that: The processing unit includes a plurality of multiplication and accumulation units.
6. A convolution calculation method, the method being applied to the processing unit according to any one of claims 1 to 5, the processing unit comprising a plurality of multiplication and accumulation units, characterized in that: The method comprises: Obtain target weight data, where the size of the target weight data is the same as the size of a unit convolution kernel, where the unit convolution kernel is the minimum size of a convolution kernel used for convolution calculation; Acquire each target non-zero element in the target weight data and position information of each of the target non-zero elements; The target non-zero elements are serially fed into the target multiplication-accumulation units in the order of the position information, and each of the target multiplication-accumulation units is used to calculate the multiplication-accumulation operation of at least one row of elements in the target weight data; Selecting target activation data corresponding to the target non-zero element to enter the target multiplication-accumulation unit; Performs a multiply-accumulate operation.
7. The convolution calculation method according to claim 6, characterized in that: The steps to perform a multiply-accumulate operation are: Calculate the product of the current target non-zero element and its corresponding target activation data as the current product; Determine whether the difference between the row coordinate value corresponding to the current target non-zero element and the row coordinate value of the previous target non-zero element is zero; If so, add the current product to the previous product; If not, the moving position of the current product is determined according to the difference between the row coordinates corresponding to the current target non-zero element and the row coordinates of the previous target non-zero element.
8. A convolution computing device, the device being applied to the processing unit according to any one of claims 1 to 5, the processing unit comprising a plurality of multiplication and accumulation units, characterized in that: The device comprises: A first acquisition module is used to acquire target weight data, where the size of the target weight data is the same as the size of the unit convolution kernel; A second acquisition module is used to acquire each target non-zero element in the target weight data and position information of each target non-zero element; An input module, used for serially inputting the target non-zero elements into a target multiplication-accumulation unit according to the sequence of the position information, each of the target multiplication-accumulation units being used for calculating the multiplication-accumulation operation of at least one row of elements in the target weight data; A selection module, used for selecting target activation data corresponding to the target non-zero element to enter the target multiplication-accumulation unit; The multiply-accumulate module is used to perform multiply-accumulate operations.
9. The convolution computing device according to claim 8, characterized in that: The multiplication-accumulation module comprises: A calculation unit, used for calculating the product of the current target non-zero element and its corresponding target activation data as the current product; A judging unit, used to judge whether the difference between the row coordinate value corresponding to the current target non-zero element and the row coordinate value of the previous target non-zero element is zero; an accumulating unit, configured to add a current product to a previous product when the judging unit determines that the difference between the row coordinate value corresponding to the current target non-zero element and the row coordinate value of the previous target non-zero element is zero; A moving unit is used to determine the moving position of the current product according to the difference between the row coordinates corresponding to the current target non-zero element and the row coordinates of the previous target non-zero element when the judgment unit determines that the difference between the row coordinates corresponding to the current target non-zero element and the row coordinates of the previous target non-zero element is not zero.
