Sparse convolutional neural network accelerator, data processing method and medium

Abstract

The invention relates to a sparse convolutional neural network accelerator, a data processing method and a medium. The accelerator architecture comprises a convolution accelerator module, a weight compression module, a computing unit array and a load balancing module. A compression weight vector and a weight mask are generated through a weight compression technology, so that the data storage and transmission quantity is remarkably reduced; a dynamic skipping mechanism based on mask and zero value detection is introduced into a calculation unit, and invalid calculation is effectively eliminated; and a weight weighted polling and characteristic value cyclic scheduling dual load balancing strategy is adopted, so that load balancing of each unit of the calculation array is ensured, and the hardware utilization rate is greatly improved. The sparse convolutional neural network accelerator provided by the invention supports efficient sparse convolutional calculation, reduces the calculation amount of the convolutional neural network, is suitable for edge calculation, automatic driving and other scenes, the actual measurement shows that the utilization rate of a calculation unit can reach 92%, the performance is improved by 2.78 times compared with that of a traditional scheme, and meanwhile, the power consumption and storage bandwidth requirements are remarkably reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of integrated circuit design, in particular to a sparse convolutional neural network accelerator, a data processing method and a medium. BACKGROUND

[0002] A convolutional neural network is a kind of feedforward neural network, which extracts local features and hierarchical representations from input data through the hierarchical structure of convolutional layers, pooling layers and fully connected layers, gradually compresses information, reduces redundancy and improves generalization ability. The convolutional neural network was proposed in the 1980s. In 2012, AlexNet showed amazing ability in large-scale image classification problems, and the convolutional neural network has attracted extensive research. With the continuous development of convolutional neural network models and training algorithms, and the increasing computing power and large-scale data sets, its performance in various tasks is constantly improving. Today, the convolutional neural network has become one of the most important technologies in the field of deep learning. Due to its powerful feature extraction and classification ability, the convolutional neural network has been widely used in image and video recognition and classification, natural language processing, object detection and speech processing, etc. Although the improvement of the size of the convolutional neural network makes it have excellent accuracy, at the same time, the complex structure, large weight parameters and large amount of calculation put high requirements on the computing power and memory of the implementation platform, and bring serious challenges to the design of hardware accelerators for model deployment.

[0003] Researchers found that the weight parameters and feature map parameters of the algorithm model both have the characteristics of sparsity by analyzing the characteristics of the convolutional neural network algorithm. The sparsity of the weight mainly comes from the pruning optimization technology, which has been proved to effectively compress the model while ensuring the accuracy of the neural network. Pruning removes redundant connections in neurons by evaluating the importance of neurons or weights, achieving the purpose of compressing the network model. The sparsity of the feature map mainly comes from the activation function ReLU, because ReLU sets all non-positive values in the intermediate results of convolution operation to 0. Sparsity means that a large number of "0" values will participate in invalid operation during forward inference. Convolutional neural network mainly contains multiply-add operation, and 0 multiplied by any number is 0, so the part of 0 value participating in the multiply-add operation can be considered as invalid operation. If these invalid operations can be removed, the computational complexity of the entire network can be greatly reduced.

[0004] When the sparsity of the weight and the eigenvalue is utilized, the problem of uneven load of the calculation unit may be encountered due to the irregularity and uneven distribution of the sparse data. In the actual calculation process, the data processed by different calculation units often have different degrees of sparsity, thereby causing different operation times. The calculation unit with a short operation time needs to wait for the calculation unit with a long operation time to complete the operation before the next stage of operation can be performed, the overall hardware utilization is low, and the improvement of the calculation efficiency and performance is hindered. SUMMARY

[0005] To solve the above technical problems, the sparse convolutional neural network accelerator provided by the application comprises:

[0006] The convolution accelerator module comprises a main controller, a direct memory access module (DMA), an input feature map cache, a convolution kernel cache, an output feature map cache, an array of calculation units, an accumulation module, and an activation and pooling module.

[0007] The weight compression module is configured to convert the original weight vector into a compressed weight vector and a corresponding weight mask, wherein each bit of the weight mask indicates whether the corresponding weight in the original weight vector is zero.

[0008] The calculation unit is integrated in the array of calculation units and comprises an eigenvalue selection module, a comparator and a multiplier-accumulator. The eigenvalue selection module selects the eigenvalue corresponding to the non-zero weight from the input eigenvalue according to the weight mask. The comparator judges whether the selected eigenvalue is non-zero and generates a control signal. The multiplier-accumulator skips the multiplication and accumulation operation of the zero-value eigenvalue in response to the control signal.

[0009] The load balancing module executes a weight weighted round robin distribution strategy and an eigenvalue cyclic scheduling strategy to balance the calculation load of each row and each column in the array of calculation units.

[0010] In an embodiment of the application, the main controller is configured to parse external instructions containing the layer type, the input feature map size, the convolution kernel size and the convolution step size, and generate data read-write addresses and control signals based on the instructions to schedule the complete convolution calculation process.

[0011] In an embodiment of the application, the calculation unit further comprises an eigenvalue register, a weight mask register, a weight FIFO and a partial sum register. The weight FIFO is configured to store the compressed weight vector, and the operation result of the multiplier-accumulator is stored in the partial sum register as an intermediate result for accumulation.

[0012] In an embodiment of the application, the weight weighted round robin distribution strategy comprises: sorting the weight vectors according to the number of non-zero values, and distributing them to each column of the array of calculation units by using a weighted round robin algorithm, so that the total number of non-zero weights processed by each column tends to be balanced.

[0013] In one embodiment of the present invention, the eigenvalue cyclic scheduling strategy includes: controlling each row of the computing unit array to cyclically load different eigenvalue vectors in a fixed order, and staggering the initial loading vectors of different rows, so that all computing units process the same total number of non-zero eigenvalues ​​within the computing cycle.

[0014] In one embodiment of the present invention, the accumulation module uses an adder array structure to perform parallel accumulation of multiple parts from the computing unit array and intermediate results in the accumulation module RAM, and after multiple rounds of accumulation calculation, the final convolution result is obtained.

[0015] In one embodiment of the present invention, the input feature map cache adopts a multi-level structure or row buffer design to improve the reuse rate of the input feature map in the convolutional sliding window operation.

[0016] In one embodiment of the present invention, the output control signal of the comparator is the result of judging that the feature value corresponding to the non-zero weight is non-zero; the multiply-accumulate unit performs multiply-accumulate operation only when the corresponding feature value is non-zero.

[0017] The present invention also provides a data processing method for a sparse convolutional neural network accelerator, comprising the following steps:

[0018] Step S1: Compress the original weights using the weight compression module to generate a compressed weight vector and a weight mask;

[0019] Step S2: Through the load balancing module, data is distributed to the computing unit array using a weighted round-robin strategy and a feature value cyclic scheduling strategy;

[0020] Step S3: In the calculation unit, select the feature value according to the weight mask and skip the calculation of zero feature value;

[0021] Step S4: Accumulate the partial sums generated by the calculation, and output the result after activation and pooling operations.

[0022] The present invention also provides an electronic computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data processing method described above.

[0023] Compared with the prior art, the above-mentioned technical solution of the present invention has the following advantages: The sparse convolutional neural network accelerator and data processing method described in the present invention effectively eliminate invalid computations in sparse data through innovative weight compression design and zero-value skipping mechanism in the computing unit. This design only performs multiplication and addition operations on non-zero weights and feature values, fundamentally and significantly reducing the total amount of computation. Secondly, the original load balancing module solves the problem of low hardware utilization caused by uneven distribution of sparse data through a dual strategy of weight-weighted round-robin and feature-value cyclic scheduling. This makes the workload of each processing unit in the computing unit array tend to be balanced, avoiding idle waiting, thereby maximizing the overall computing efficiency and throughput. Furthermore, this architecture combines high performance and versatility. It not only significantly improves the computing speed of sparse models and reduces system power consumption and storage bandwidth requirements, but also flexibly adapts to various sparse patterns, efficiently supporting mainstream pruning optimization networks without changing the hardware structure, providing an ideal hardware solution for deploying complex neural network models in mobile terminals and edge devices. Attached Figure Description

[0024] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0025] Figure 1 The block diagram of the sparse convolutional neural network accelerator provided by the present invention.

[0026] Figure 2 This is a schematic diagram of the data flow and control system of the computing unit array provided by the present invention.

[0027] Figure 3 A schematic diagram illustrating the workflow of the weight compression module provided by this invention.

[0028] Figure 4 This is a diagram of the internal structure of the computing unit provided by the present invention.

[0029] Figure 5 This is a timing example of a computing unit performing sparse multiply-accumulate computation provided by the present invention.

[0030] Figure 6 The input feature map and convolution kernel data structure are provided in an example of convolution calculation for this invention.

[0031] Figure 7 In a convolution calculation example provided by the present invention, the weight loading of each calculation unit is shown after applying a weighted round-robin allocation strategy.

[0032] Figure 8This invention provides an example of convolution calculation, showing the feature value loading status of each calculation unit after applying a feature value cyclic scheduling strategy. Detailed Implementation

[0033] Example 1

[0034] like Figure 1 As shown, this embodiment provides a sparse convolutional neural network accelerator, the core of which is to comprehensively solve the problems of low computational efficiency and high energy consumption of traditional accelerators when processing sparse models through sparsity-aware computing, dynamic instruction filtering, load balancing optimization and efficient data flow management.

[0035] This accelerator can be integrated into a system-on-a-chip (SoC) as an IP core. The SoC also includes a central processing unit (CPU), a network-on-a-chip (NoC), and an external memory interface. The CPU, acting as the main controller, sends instructions to the accelerator's main controller via configuration registers to initiate convolutional computation tasks. The accelerator interacts with external memory (such as DDR) via DMA to efficiently read input feature maps and weights and write back output feature maps. This embodiment of the accelerator is particularly suitable for mobile terminal devices with high power consumption and real-time performance requirements, enabling efficient image recognition and processing functions.

[0036] Specifically, the overall system architecture includes: a main controller, DMA, input feature map cache, convolution kernel cache, output feature map cache, computation unit array, accumulation module, and activation and pooling modules.

[0037] The main controller is responsible for parsing external input instructions and scheduling the entire convolutional computation process. The instruction information specifically includes layer type, activation type, pooling type, input feature map height / width, number of input channels, convolutional kernel height / width / number of kernels, convolutional stride, and zero-padding. Based on the instructions, the main controller sends the read address of the input feature map and weights, the write address of the output feature map, and generates various signals to control the computation process.

[0038] The DMA is responsible for reading feature values ​​and weight data from external storage, storing them in the input feature map cache and convolution kernel cache respectively, and writing the data in the output feature map cache back to external storage.

[0039] The computational unit array is responsible for performing convolution calculations on the input feature values ​​and weights. The accumulation module is responsible for accumulating the partial sums of the outputs of the computational unit array to obtain the final convolution result. The activation and pooling module is responsible for performing activation and pooling operations on the final convolution result to obtain the output feature map data.

[0040] Specific computing unit array data streams, such as Figure 2The diagram illustrates the data path of the computation unit array. The input feature value controller requests the necessary feature values ​​from the input feature map cache and sends them to the computation units. The weight controller requests the necessary weights from the convolution kernel cache and sends them to the computation units. The computation unit array contains multiple computation units; each row of computation units shares the feature values ​​sent by the input feature value controller, and each column of computation units shares the weights sent by the weight controller. In this embodiment, the array uses a 4×4 two-dimensional arrangement to support large-scale parallel processing.

[0041] Specific weight compression designs, such as Figure 3 The diagram illustrates the workflow of the weight compression module provided by this invention. The original weight vector in the convolution kernel is converted into a compressed weight vector and a weight mask after weight compression. The compressed weight vector is obtained by removing zero values ​​from the original weight vector. The bit width of the weight mask is equal to the number of data points in the original weight vector, and each bit of the mask corresponds one-to-one with a data point in the original weight vector. When the original weight is zero, the corresponding weight mask is 0; when the original weight is non-zero, the corresponding weight mask is 1. For example, the original vector [0, 2.5, 0, -1.3] is compressed into [2.5, -1.3] and the mask [0, 1, 0, 1]. This design significantly reduces the storage space and transmission bandwidth occupied by the weight data.

[0042] The specific internal structure of the computing unit and sparse computing, such as Figure 4 The diagram shows the internal structure design of the computing unit implemented in this invention. It includes an eigenvalue register, a weight mask register, a weight FIFO, an eigenvalue selection module, a comparator, a multiply-accumulator, and a partial sum register.

[0043] The feature selection module is responsible for selecting the feature values ​​corresponding to non-zero weights from the feature vector based on the weight mask. The comparator is responsible for determining whether the input feature value is non-zero. When the input feature value is non-zero, the comparator will pull the valid signal high; when the input feature value is zero, the comparator will pull the valid signal low. The multiply-accumulator is responsible for receiving the feature value, weight, and partial sum. It performs multiplication and accumulation when the valid signal is high and does not perform calculations when the valid signal is low, thus skipping invalid operations for feature values ​​of 0. The intermediate results obtained from the calculation are output to the partial sum register for temporary storage and accumulation.

[0044] Figure 5 The multiplication and addition process of the computation unit is illustrated with a specific example. The input feature value is "5093", the weight mask is "1101", and the compressed weight vector is "126".

[0045] In the first calculation cycle, the weight FIFO outputs the first non-zero weight "1". The feature selection module selects the corresponding feature value "5" according to the weight mask. The comparator determines that it is a non-zero value, pulls up the valid signal, and the multiply-accumulate unit performs the calculation and stores the result 5 into the partial sum register.

[0046] In the second calculation cycle, the weight FIFO outputs a weight of "2", corresponding to a feature value of "0", the comparator pulls the valid signal low, and the multiply-accumulate skips this calculation.

[0047] In the third calculation cycle, the weight FIFO outputs a weight of "6", corresponding to a feature value of "3", the valid signal is valid, the multiply-accumulator calculates 5 + 6 * 3 = 23 and updates the partial sum register.

[0048] Furthermore, this embodiment also provides a load balancing design to reduce the impact of uneven data distribution on computational efficiency. This design includes a weighted round-robin strategy and an eigenvalue-based cyclic scheduling strategy.

[0049] Figure 6 to Figure 8 The effect of the load balancing strategy is demonstrated through a convolution calculation example. Figure 6 The input feature maps (vectors A0-A3) and convolution kernels (vectors K0_0 to K15_0, sorted by the number of non-zero values) are shown.

[0050] The weighted round-robin strategy first sorts the weight vectors according to the number of non-zero values ​​from smallest to largest, and then uses a weighted round-robin algorithm to allocate the weight vectors when the computing unit loads the next weight vector.

[0051] like Figure 7 As shown (taking the first row of calculation units 00-03 as an example), calculation unit 00 first loads K0_0 with the fewest non-zero weights, and then loads K15_0 with the most non-zero weights, so that the number of non-zero weights loaded in the first two times is close to that of other units, thereby achieving load balancing of each column of calculation units.

[0052] Furthermore, computation units 00 to 03 constitute the computation units in the first row of the computation array. Due to the use of a weighted round-robin strategy, computation unit 00 first loads K0_0, which has the fewest non-zero weights, and then loads K15_0, which has the most non-zero weights. This avoids both excessive workload due to loading K0_0 and excessive workload due to loading K15_0. The number of non-zero weights loaded in the first two loads is close to the number loaded in the first two loads of the other three computation units. Since the number of non-zero weights loaded in the first two loads of each computation unit is close to the number loaded in the last two loads, the number of non-zero weights loaded by the four computation units is balanced during one round of convolution computation. The loading situation of the remaining three rows of computation units is the same as that of the first row.

[0053] The eigenvalue cyclic scheduling strategy is as follows: during the convolution calculation process, each row of computing units in the computing unit array cyclically loads eigenvalue vectors in the order of eigenvalue vectors, and the eigenvalue vectors loaded for the first time in different rows are different.

[0054] like Figure 8 As shown (taking the first column of calculation units 00-30 as an example), during the first loading, calculation units 00, 10, 20, and 30 load A0, A1, A2, and A3 respectively, and then load them sequentially in a loop. This ensures that each calculation unit loads all eigenvalue vectors in one round of calculation, thus processing the same number of non-zero eigenvalues.

[0055] Furthermore, computation units 00 to 30 constitute the first column of the computation array. Due to the eigenvalue cyclic scheduling strategy, during the first loading, computation unit 00 loads A0, computation unit 10 loads A1, computation unit 20 loads A2, and computation unit 30 loads A3. Subsequently, each computation unit cyclically loads the eigenvalue vector in the order of "A0-A1-A2-A3". During one round of convolution computation, each computation unit loads A0, A1, A2, and A3, therefore the number of non-zero eigenvalues ​​loaded is also equal. The loading of the remaining three columns of computation units is the same as that of the first column.

[0056] Example 2

[0057] This embodiment details the software co-verification process of the data processing method for the sparse convolutional neural network accelerator. During the chip design phase, a co-simulation platform containing the accelerator's RTL model and CPU model is built. The driver running on the CPU executes the data processing method steps: first, the compressed model weights and mask are loaded into the accelerator; then, load balancing strategy parameters are configured and computation is started; finally, polling waits for the accelerator to complete computation and then interrupts.

[0058] In terms of data flow management, this embodiment has undergone in-depth optimization. The input feature map cache adopts a ping-pong cache structure, which uses a double buffering mechanism to prefetch the next data block while computing the current data block, effectively hiding data access latency. Data is injected into the computing array through a pulsating input layer to maximize data reuse.

[0059] Technical performance verification shows that, under sparse models such as VGG16, this accelerator reduces computation time by 7.29% compared to designs without load balancing. In broader tests (such as sparse AlexNet), it achieves a 2.78x performance improvement compared to traditional GPUs (cuSPARSE), with a computational unit utilization of up to 92%, and saves approximately 60% of memory bandwidth by eliminating invalid computations. This design is particularly suitable for edge computing scenarios such as autonomous driving, drone vision processing, and 3D point cloud analysis, providing an efficient hardware solution for achieving high-energy-efficiency sparse neural network inference.

[0060] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A sparse convolutional neural network accelerator, characterized in that, include: The convolution accelerator module includes a main controller, a direct memory access (DMA) module, an input feature map cache, a convolution kernel cache, an output feature map cache, a computational unit array, an accumulation module, and activation and pooling modules; The weight compression module is used to convert the original weight vector into a compressed weight vector and a corresponding weight mask, wherein each bit of the weight mask indicates whether the corresponding weight in the original weight vector is zero; The computing unit, integrated in the computing unit array, includes a feature value selection module, a comparator, and a multiply-accumulator; wherein, the feature value selection module selects the feature value corresponding to the non-zero weight from the input feature values ​​according to the weight mask, the comparator determines whether the selected feature value is non-zero and generates a control signal, and the multiply-accumulator responds to the control signal and skips the multiplication and addition operation of zero-value feature values; The load balancing module executes a weighted round-robin allocation strategy and a feature value cyclic scheduling strategy to balance the computing load of each row and column in the computing unit array.

2. The sparse convolutional neural network accelerator according to claim 1, characterized in that, The main controller is configured to: parse external instructions containing layer type, input feature map size, convolution kernel size and convolution stride, and generate data read / write addresses and control signals based on the instructions to schedule the complete convolution calculation process.

3. The sparse convolutional neural network accelerator according to claim 1, characterized in that, The calculation unit further includes a feature value register, a weight mask register, a weight FIFO, and a partial sum register; the weight FIFO is used to store the compressed weight vector, and the operation result of the multiply-accumulate is stored in the partial sum register as an intermediate result for accumulation.

4. The sparse convolutional neural network accelerator according to claim 1, characterized in that, The weighted round-robin allocation strategy includes: sorting the weight vectors according to the number of non-zero values, and using a weighted round-robin algorithm to allocate them to each column of the computing unit array, so that the total number of non-zero weights processed by each column tends to be balanced.

5. The sparse convolutional neural network accelerator according to claim 1, characterized in that, The eigenvalue cyclic scheduling strategy includes: controlling each row of the computing unit array to cyclically load different eigenvalue vectors in a fixed order, and staggering the initial loading vectors of different rows, so that all computing units process the same total number of non-zero eigenvalues ​​within the computing cycle.

6. The sparse convolutional neural network accelerator according to claim 1, characterized in that, The accumulation module uses an adder array structure to perform parallel accumulation of multiple parts from the computing unit array and intermediate results in the accumulation module RAM. After multiple rounds of accumulation calculation, the final convolution result is obtained.

7. The sparse convolutional neural network accelerator according to claim 1, characterized in that, The input feature map cache adopts a multi-level structure or row buffer design to improve the reuse rate of input feature maps in convolutional sliding window operations.

8. The sparse convolutional neural network accelerator according to claim 1, characterized in that, The output control signal of the comparator is the result of judging that the feature value corresponding to the non-zero weight is non-zero; the multiply-accumulate unit only performs multiply-accumulate operation when the corresponding feature value is non-zero.

9. A data processing method for a sparse convolutional neural network accelerator, applied to the sparse convolutional neural network accelerator of any one of claims 1 to 8, characterized in that, Includes the following steps: Step S1: Compress the original weights using the weight compression module to generate a compressed weight vector and a weight mask; Step S2: Through the load balancing module, data is distributed to the computing unit array using a weighted round-robin strategy and a feature value cyclic scheduling strategy; Step S3: In the calculation unit, select the feature value according to the weight mask and skip the calculation of zero feature value; Step S4: Accumulate the partial sums generated by the calculation, and output the result after activation and pooling operations.

10. An electronic computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the data processing method as described in claim 9.

Images