A neural network acceleration method, device, equipment and storage medium

Abstract

The embodiment of the present application discloses a neural network acceleration method, apparatus, device and storage medium, the method comprising: determining a first convolution kernel of a neural network structure; wherein the number of non-zero weight data of each input channel in the first convolution kernel is the same; centrally storing the non-zero weight data of each input channel in the first convolution kernel to obtain a second convolution kernel of the neural network structure; performing a convolution operation with the feature image using the second convolution kernel to obtain a target convolution result. In this way, since the first convolution kernel determined in the present application only requires the number of non-zero weight data of each input channel to be the same, the degree of freedom of algorithm pruning is increased, and the effect loss caused by pruning is reduced; in addition, the centralized storage strategy for non-zero weight data can not only improve the utilization rate of hardware resources, but also improve the energy efficiency ratio of the algorithm implementation and reduce the delay of the algorithm implementation while ensuring the algorithm effect.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a neural network acceleration method, apparatus, device and storage medium. Background Art

[0002] In recent years, the rise of deep learning (DL) has continuously driven the development of artificial intelligence. DL algorithms, represented by convolutional neural networks, have been widely used in fields such as computer vision and autonomous driving for tasks such as classification, recognition, noise reduction, and super-resolution. However, their complex structures severely restrict their deployment on power- and resource-constrained mobile and embedded platforms, necessitating a degree of compression.

[0003] Currently, pruning can be implemented at the algorithmic level, using hardware architecture to add a bypass module before the multiplier to skip zero-value operations. However, this approach requires adding logic before the multiplier to determine whether the input to the next operation contains a zero value. If the input contains a zero value, the result is directly set to 0. In this case, the multiplier is not involved in the actual operation, resulting in a waste of resources. Summary of the Invention

[0004] This application proposes a neural network acceleration method, device, equipment and storage medium, which can improve the utilization rate of hardware resources, and while ensuring the algorithm effect, it can also improve the energy efficiency ratio of the algorithm implementation and reduce the delay of the algorithm implementation.

[0005] To achieve the above objectives, the technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a neural network acceleration method, applied to an electronic device, the method comprising:

[0007] Determine a first convolution kernel of the neural network structure; wherein the number of non-zero weight data of each input channel in the first convolution kernel is the same;

[0008] The non-zero weight data of each input channel in the first convolution kernel are centrally stored to obtain the second convolution kernel of the neural network structure;

[0009] The second convolution kernel is used to perform a convolution operation with the feature image to obtain the target convolution result.

[0010] In a second aspect, an embodiment of the present application provides a neural network acceleration device, which is applied to an electronic device. The neural network acceleration device includes a determination unit, a storage unit, and a calculation unit; wherein,

[0011] A determining unit configured to determine a first convolution kernel of the neural network structure; wherein the number of non-zero weight data of each input channel in the first convolution kernel is the same;

[0012] A storage unit configured to centrally store non-zero weight data of each input channel in the first convolution kernel to obtain a second convolution kernel of the neural network structure;

[0013] The computing unit is configured to perform a convolution operation on the feature image using the second convolution kernel to obtain a target convolution result.

[0014] In a third aspect, an embodiment of the present application provides a chip comprising the neural network acceleration device as described in the second aspect.

[0015] In a fourth aspect, an embodiment of the present application provides an electronic device, the electronic device including a memory and a processor; wherein,

[0016] a memory for storing computer programs capable of running on the processor;

[0017] A processor is configured to execute the method of the first aspect when running a computer program.

[0018] In a fifth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program, and when the computer program is executed by a processor, implements the method of the first aspect.

[0019] The embodiments of the present application provide a neural network acceleration method, apparatus, device and storage medium, which are applied to electronic devices to determine the first convolution kernel of the neural network structure; wherein, the number of non-zero weight data of each input channel in the first convolution kernel is the same; the non-zero weight data of each input channel in the first convolution kernel are centrally stored to obtain the second convolution kernel of the neural network structure; the second convolution kernel is used to perform a convolution operation with the feature image to obtain the target convolution result. In this way, since the first convolution kernel determined by the present application only requires the number of non-zero weight data of each input channel to be the same, the degree of freedom of algorithm pruning is increased, and the effect loss caused by pruning is reduced; in addition, the centralized storage strategy for non-zero weight data can not only improve the utilization rate of hardware resources, but also improve the energy efficiency ratio of the algorithm implementation while ensuring the algorithm effect, thereby reducing the delay of the algorithm implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of the convolution calculation process;

[0021] Figure 2 A schematic diagram of a fine-grained pruning process;

[0022] Figure 3A schematic diagram of a coarse-grained pruning process;

[0023] Figure 4 A flowchart of a neural network acceleration method provided in an embodiment of the present application;

[0024] Figure 5 A flowchart of another neural network acceleration method provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of a pruning process provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of a method for storing pruned weight data provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of a method for encoding a position index value provided in an embodiment of the present application;

[0028] Figure 9 A schematic diagram of a process for obtaining corresponding feature data in a feature image according to the position of non-zero weight data provided in an embodiment of the present application;

[0029] Figure 10 A schematic diagram of a computing array provided in an embodiment of the present application;

[0030] Figure 11 A flowchart of another neural network acceleration method provided in an embodiment of the present application;

[0031] Figure 12 A schematic diagram of the hardware data scheduling sequence for the first clock cycle (time t0) provided in an embodiment of the present application;

[0032] Figure 13 A schematic diagram of a hardware data scheduling sequence for the second clock cycle (time t1) provided in an embodiment of the present application;

[0033] Figure 14 A schematic diagram of a hardware data scheduling sequence for the third clock cycle (time t2) provided in an embodiment of the present application;

[0034] Figure 15 A schematic diagram of convolution calculation based on a centralized storage method after pruning provided in an embodiment of the present application;

[0035] Figure 16 A schematic diagram of the structure of a neural network acceleration device provided in an embodiment of the present application;

[0036] Figure 17 A schematic diagram of the specific hardware structure of an electronic device provided in an embodiment of the present application;

[0037] Figure 18 A schematic diagram of the structure of a chip provided in an embodiment of the present application;

[0038] Figure 19 A schematic diagram of the specific hardware structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0041] In the following description, reference is made to "some embodiments," which describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first, second, and third" in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first, second, and third" may be interchanged in a specific order or sequential order where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0042] In recent years, with the development of computer technology, machine learning using neural networks has become a research hotspot. Deep learning (DL), in particular, is a new research direction in the field of machine learning. It was introduced to machine learning to bring it closer to its original goal: artificial intelligence (AI). Among them, the convolutional neural network (CNN) is a type of neural network. It is a feedforward neural network (FNN) with a deep structure that includes convolutional calculations and is one of the representative algorithms of deep learning.

[0043] It is understandable that the basic operation of convolution is the product of two tensors, see Figure 1 .like Figure 1The weight data at the same position is multiplied with the data of the feature image, and then all the product results are accumulated to obtain the final convolution result of the two tensors. There are a certain number of zero values in the weight data (such as Figure 1 Since zero multiplied by any number is zero, it is meaningless to use these zero values in calculations, which not only wastes a lot of computing units but also wastes transmission bandwidth.

[0044] In the related art, for neural network models, the model operation requires a lot of computing, memory and power consumption. In order to solve the bottleneck of the model operation, a series of model compression methods can be used, including model pruning (Pruning), which sets some unimportant weight data to zero. In addition, a technology can also be used in the design of neural network model accelerators. If there is zero in the multiplier, then the result must be zero, so through a certain method, it can skip the operation with zero multiplier (Zero Skipping), which is also beneficial for saving power consumption. In this way, the current technical solution can implement pruning operations at the algorithm level. Pruning can be divided into coarse-grained pruning and fine-grained pruning, and the hardware architecture can implement skipping zero value operations through zero skipping technology or adding bypass.

[0045] In a specific example, see Figure 2 , which shows a schematic diagram of the fine-grained pruning process. Figure 2 As shown in the figure, there are four convolution kernels, K0, K1, K2, and K3. The grid lines represent zero values, while the gray lines represent non-zero values. The original convolution kernels contain a large number of non-zero values. To enable the neural network model to be deployed on the device side, the model needs to be compressed and pruned. For example, fine-grained pruning forcibly sets non-zero values close to zero. Therefore, after fine-grained pruning, the distribution of zero values becomes irregular. While this approach offers greater flexibility for upper-layer algorithms, simpler pruning, and minimal impact on algorithm performance, the resulting zero value distribution after pruning is too random, making it difficult for hardware implementation and significantly increasing hardware design complexity. For hardware, a well-organized data structure is more conducive to efficient convolution.

[0046] In another specific example, see Figure 3 , which shows a schematic diagram of the coarse-grained pruning process. Figure 3 As shown in the figure, in coarse-grained pruning, all the values of some convolution kernels are directly set to zero according to the distribution of non-zero values in the convolution kernel. In this way, the convolution kernel does not participate in the calculation in the hardware. However, the pruning granularity of this method is too large, which is not good for the flexibility of the algorithm layer and will also affect the effect of the algorithm.

[0047] In hardware design, it's also possible to skip zero-value operations by adding a bypass module before the multiplier. However, this approach requires adding judgment logic before the multiplier to determine in real time whether the input to the next operation contains a zero value. If the input contains a zero value, the result is simply set to zero. Although the multiplier is not involved in the actual operation, it also does not participate in other valid value calculations, resulting in a waste of resources. Based on this, the current technical difficulty lies in finding a suitable method to fully integrate algorithm pruning and hardware architecture zero-skipping technology within a limited circuit.

[0048] The embodiment of the present application provides a neural network acceleration method, the basic idea of which is: determining the first convolution kernel of the neural network structure; wherein the number of non-zero weight data of each input channel in the first convolution kernel is the same; the non-zero weight data of each input channel in the first convolution kernel are centrally stored to obtain the second convolution kernel of the neural network structure; the second convolution kernel is used to perform a convolution operation with the feature image to obtain the target convolution result. In this way, since the first convolution kernel determined by the present application only requires the number of non-zero weight data of each input channel to be the same, the freedom of algorithm pruning is increased and the effect loss caused by pruning is reduced; in addition, the centralized storage strategy for non-zero weight data can not only improve the utilization rate of hardware resources, but also improve the energy efficiency ratio of the algorithm implementation while ensuring the algorithm effect, thereby reducing the delay of the algorithm implementation.

[0049] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0050] In one embodiment of the present application, see Figure 4 , which shows a flow chart of a neural network acceleration method provided by an embodiment of the present application. Figure 4 As shown, the method may include:

[0051] S401: Determine a first convolution kernel of a neural network structure; wherein the number of non-zero weight data of each input channel in the first convolution kernel is the same.

[0052] It should be noted that the execution subject of the embodiments of the present application is a neural network acceleration device, or an electronic device integrated with the device. The electronic device may be a smartphone, tablet computer, laptop computer, PDA, PMP, navigation device, wearable device, etc., without any limitation here.

[0053] It should also be noted that the neural network structure described herein may refer to a convolutional neural network structure; or, alternatively, a pre-trained network structure, which may include multiple convolutional layers, each of which has a convolution kernel and its corresponding weight data. For each convolution kernel, the weight data may include non-zero weight data and zero-valued weight data, i.e., non-zero and zero values.

[0054] In an embodiment of the present application, the first convolution kernel can be obtained by pruning. Figure 5 FIG. 1 shows a flow chart of another neural network acceleration method provided by an embodiment of the present application. Figure 5 As shown, step S401 may include:

[0055] S501: Obtain the initial convolution kernel of the neural network structure.

[0056] S502: Perform a pruning operation on each input channel in the initial convolution kernel to obtain a first convolution kernel of a neural network structure; wherein the number of non-zero weight data of each input channel in the first convolution kernel is the same.

[0057] That is to say, in order to achieve better algorithmic effects, a fine-grained pruning method can be used, but the distribution of its weight data is irregular; while a coarse-grained pruning method is more hardware-friendly, it results in a greater loss of effect. In an embodiment of the present application, a more balanced pruning method is proposed, which can take into account the friendliness of hardware implementation while leaving the pruning algorithm with small constraints and a large search space, thereby increasing the degree of freedom of pruning. That is, the pruning strategy of the embodiment of the present application only stipulates that the number of non-zero weight data of each input channel in the convolution kernel is the same.

[0058] For example, assume that there are two convolution kernels K0 and K1, and the distribution of their 0 values is random. The pruning strategy that can be adopted in the embodiment of the present application is to stipulate that the number of non-zero values in each input channel (which can be represented by IC) of each convolution kernel is the same. Figure 6 As shown in the figure, for convolution kernels K0 and K1, after pruning, the number of non-zero values in each input channel of K0 and K1 is 5; non-zero values are filled with gray, and zero values are filled with network lines. In other words, the distribution of zero values in each input channel can be random, but the number of zero values in each input channel is the same (such as Figure 6 The number of 0 values shown is 4).

[0059] In the embodiment of the present application, taking the convolution kernel size of 3×3 as an example, the number of 0 values in each input channel can be any value between 0 and 9, and the number of non-0 values in each input channel can also be any value between 0 and 9, but the sum of the number of 0 values and the number of non-0 values is 9. Among them, the specific number of 0 values or non-0 values is set according to the actual situation. The embodiment of the present application can be explored through the algorithm, and the constraints on the upper-level algorithm are smaller, which is more conducive to exploring pruning solutions with less effect loss.

[0060] S402: Centrally store the non-zero weight data of each input channel in the first convolution kernel to obtain a second convolution kernel of the neural network structure.

[0061] It should be noted that after pruning to determine the first convolution kernel, considering the complexity of hardware implementation, the embodiment of the present application can implement the following hardware storage solution: first, it is necessary to record the positions of all non-zero weight data of each input channel in the first convolution kernel, and then place these non-zero weight data in a centralized location of the first convolution kernel for storage.

[0062] Furthermore, since the position of the non-zero weight data has changed after the centralized location storage, it is necessary to open a new storage unit to store the position index value of the non-zero weight data in each input channel. In some embodiments, for S402, when the non-zero weight data of each input channel in the first convolution kernel is centrally stored, the method may further include:

[0063] Determine the position index value corresponding to the non-zero weight data of each input channel;

[0064] The position index value is stored in a preset storage area of each input channel; wherein the position index value is used to indicate the position of the non-zero weight data in each input channel.

[0065] For example, Figure 7 For example, after the number of non-zero weight data of each input channel in the first convolution kernel is obtained by pruning, it is necessary to first record the positions of all non-zero values of each input channel in the convolution kernels K0 and K1, and then store these non-zero values in the centralized position of the convolution kernels K0 and K1. Since the position of the non-zero value has changed after the centralized position storage, it is necessary to open a new storage unit (such as Figure 7 The preset storage area shown is used to store the position index value of the non-zero weight data in each input channel (IC), so that the position of the non-zero value in each IC can be recorded.

[0066] In addition, for the position index value, its encoding method can be used Figure 8The encoding method shown can also be other customized methods, which are not limited in the present application embodiment. In a specific example, Figure 7 The position index value stored in is the value obtained using Figure 8 The encoding method shown.

[0067] S403: Perform a convolution operation on the feature image using the second convolution kernel to obtain a target convolution result.

[0068] It should be noted that after the centralized storage, a convolution operation can be performed based on the second convolution kernel and the feature image after the centralized storage. Specifically, in some embodiments, the convolution operation using the second convolution kernel and the feature image to obtain the target convolution result may include:

[0069] Based on the non-zero weight data of each input channel in the second convolution kernel, read the position index value of each input channel from the preset storage area of each input channel respectively;

[0070] Based on the feature image, obtain the feature data corresponding to the position index value of each input channel;

[0071] The non-zero weight data and the feature data are multiplied to obtain the multiplication result, and all the multiplication results are accumulated to obtain the target convolution result.

[0072] For example, Figure 9 For example, assuming that the second convolution kernel is represented by K0, a convolution operation is performed on K0 and the feature image. Since all non-zero values are stored in the convolution kernel and the position of the non-zero values in each input channel is recorded, the corresponding feature data can be obtained from the tensor data of the feature image according to the position of the non-zero values in each input channel recorded in the convolution kernel (for example: 0 2 4 6 7; 2 4 5 6 7; 0 2 3 4 6; 0 3 4 6 8), and a multiplication operation is performed. Then, all the products are accumulated to obtain the target convolution result of K0 and this 3×3 feature image window. Since the non-zero values of each input channel (IC) in the weight data are stored in a centralized manner, in this embodiment of the application, only 5 data reading operations are required, and the multiplication and addition calculation with the corresponding data of the feature image can be completed. In the related art, if the pruning method described in the embodiment of the present application is not implemented for the weight data and the non-zero weight data is not stored centrally, then it takes 9 data reading operations (i.e., 9 clock cycles) to complete the convolution of K0 and the feature image 3×3 window. Figure 9 The example shown requires only five data read operations (ie, five clock cycles) to complete.

[0073] In short, in the embodiments of the present application, it is necessary to take into account the friendliness of hardware implementation, give the algorithm sufficient flexibility and ensure the effect, and enable the hardware architecture to process efficiently. Therefore, a joint optimization method that integrates the upper-layer algorithm pruning and the zero-skipping technology in the hardware architecture is proposed here. At this time, the 0 value can be ignored, and only valid data (i.e., non-zero weight data) is allowed to participate in the calculation, which greatly reduces the effective computing power required for the algorithm implementation.

[0074] The embodiment of the present application provides a neural network acceleration method, which determines a first convolution kernel of a neural network structure; wherein the number of non-zero weight data of each input channel in the first convolution kernel is the same; the non-zero weight data of each input channel in the first convolution kernel are centrally stored to obtain a second convolution kernel of the neural network structure; and the second convolution kernel is used to perform a convolution operation with a feature image to obtain a target convolution result. In this way, since the first convolution kernel determined by the present application only requires the number of non-zero weight data of each input channel to be the same, the degree of freedom of algorithm pruning is increased and the effect loss caused by pruning is reduced; in addition, the centralized storage strategy for non-zero weight data not only improves the utilization rate of hardware resources, but also improves the energy efficiency ratio of the algorithm implementation while ensuring the algorithm effect, thereby reducing the delay of the algorithm implementation.

[0075] In another embodiment of the present application, the neural network acceleration method provided in the above embodiment can be implemented using a hardware architecture. Specifically, the electronic device may include a computing array consisting of m×n processing units (PEs), where m and n are integers greater than zero.

[0076] In an embodiment of the present application, each PE may include several accumulator trees, which are used to multiply the received non-zero weight data and the corresponding feature data, and accumulate all multiplication results obtained within the accumulator tree.

[0077] For example, see Figure 10 , which shows a schematic diagram of a calculation array provided by an embodiment of the present application. Among them, for conventional convolution calculation, its basic operation is multiplication and accumulation operation (including multiplication calculation and accumulation calculation), such as Figure 10 The convolutional arithmetic unit (CLU) computational array diagram shows m PEs horizontally and n PEs vertically. The horizontal direction transmits feature images in a pulsating manner, while the vertical direction corresponds to the input channels of feature images and weight data. Each PE can be cascaded, and multiplication and accumulation operations are performed internally. During the operation of this computational array, each PE can complete multiple multiplication and accumulation operations in parallel. Figure 10The left half of the figure is the magnified PE structure. The feature image is propagated horizontally in a pulsating manner. For each PE, the received non-zero weight data (represented by W) is multiplied by the corresponding feature data in the feature image (represented by F) and the accumulation is performed in the vertical direction. In other words, from an algorithmic perspective, the multiplication and accumulation operation of the convolution needs to be accumulated in the depth direction of the feature image and the convolution kernel. Therefore, it can be considered that each PE can complete the accumulation of the feature image and the convolution kernel in the depth direction (i.e., the y-axis direction). Similarly, for the horizontal direction of the computing array, it can be the expansion of the convolution kernel. That is, in the computing array, different convolution kernels are used between each PE in the horizontal direction, while the same convolution kernel is used inside each PE. In this way, for the same feature image, when it is propagated horizontally between PEs, the convolution results of this feature image and different convolution kernels are generated.

[0078] It should be noted that “transmitting the characteristic image by pulsation” is still based on Figure 10 For example, assuming that feature images flow from right to left, and each PE can process L windows of feature image data, then horizontally, after completing processing in the current PE, the feature images of these L windows will continue to propagate leftward to the adjacent PE on the left as input data, and after processing is completed, they will continue to flow leftward. Because the convolution kernel stored in each PE is different, each transmission is equivalent to performing a convolution operation on the feature images of the L windows with a different convolution kernel.

[0079] It is also important to note that in actual AI algorithm networks, weight data is distributed with zero values to varying degrees during computation. If left unprocessed, these zero values will also participate in the multiplication and accumulation operations, resulting in wasted computing units and slowing down overall operational efficiency. Based on this, the present application proposes a method for collaboratively optimizing the hardware and software architecture by combining algorithm pruning with zero-skipping technology.

[0080] In the embodiment of the present application, it is first necessary to perform a pruning operation on the weight data. Since the weight data is generated offline, the pruning strategy of the embodiment of the present application can be executed on the weight data to make the number of non-zero weight data of each input channel (IC) in each convolution kernel the same. In this way, this pruning strategy can not only increase the degree of freedom of pruning, explore more effective pruning operations as much as possible, and reduce the effect loss caused by pruning; in addition, pruning can also compress the algorithm model, and can also reduce the amount of calculation and bandwidth pressure through hardware coordination.

[0081] Furthermore, in the hardware architecture, for the convolution calculation of the multiplication and accumulation operation, Figure 11FIG. 1 shows a flow chart of another neural network acceleration method provided by an embodiment of the present application. Figure 11 As shown, the method may include:

[0082] S1101: Set a number of sliding windows; each sliding window corresponds to an accumulation tree, each accumulation tree includes C multiplication units, and C represents the number of input channels of the first convolution kernel.

[0083] S1102: Calculate an i-th calculation result corresponding to the i-th clock cycle based on a plurality of sliding windows.

[0084] S1103: When i is equal to the target clock cycle value, determine the target convolution result according to the target calculation result corresponding to the target clock cycle value; wherein i is an integer greater than 0.

[0085] It should be noted that, with respect to determining the target clock cycle value, in some embodiments, the method may further include: determining the number of non-zero weight data for each input channel in the first convolution kernel; and then determining the target clock cycle value based on the number of non-zero weight data.

[0086] In a specific example, Figure 6 For example, after pruning, the number of non-zero weight data in each input channel is 5, so it can be determined that the target clock cycle value is equal to 5.

[0087] It should also be noted that, for S1102, calculating the i-th calculation result corresponding to the i-th clock cycle based on a plurality of sliding windows may include:

[0088] When i is equal to 1, in the first clock cycle, determine the first feature data of each sliding window on each input channel; use the accumulator tree corresponding to each sliding window to multiply the first feature data on each input channel with the first non-zero weight data corresponding to each input channel in the second convolution kernel through C multiplication units, and accumulate the multiplication results corresponding to the C multiplication units through the accumulator tree to obtain a first calculation result;

[0089] When i is greater than 1, in the i-th clock cycle, the i-th feature data of each sliding window on each input channel is determined; using the accumulator tree corresponding to each sliding window, the i-th feature data on each input channel is multiplied by the i-th non-zero weight data corresponding to each input channel in the second convolution kernel through C multiplication units, and the multiplication results corresponding to the C multiplication units are accumulated with the i-1-th calculation result obtained last time through the accumulator tree to obtain the i-th calculation result.

[0090] For example, assume that L sliding windows (also called "convolution windows") are arranged horizontally. For ease of description, it can be assumed that the step length of each sliding window is 3. Here, the step length specifically refers to the interval between two adjacent sliding windows. Normally, the step length can be set to 1 or 2, but in this case, adjacent sliding windows may have overlapping pixels; therefore, in this embodiment of the present application, for ease of description, the step length can be set to 3.

[0091] See also Figure 12 , which shows a schematic diagram of the hardware data scheduling sequence of the first clock cycle (t0 moment) provided by an embodiment of the present application. Figure 12 As shown, the step length of each sliding window is 3, so that every two sliding windows are adjacent to each other without overlap, and the size of each sliding window is 3×3. Figure 10 For example, a PE in the , which contains L cumulative trees, each cumulative tree corresponds to Figure 12 A sliding window in the , the task of each accumulator tree is to calculate the convolution operation of a sliding window and a convolution kernel, Figure 12 Only the first clock cycle is described, and the corresponding feature data in the feature image that needs to be filled in the accumulation tree, such as each group of blocks filled with diagonal lines covered by different types of diagonal lines (referred to as "diagonal blocks"), and the diagonal blocks covered by different types of diagonal lines need to be mapped to different accumulation trees. There are a total of C multiplication units inside the PE, corresponding to the C feature data in the direction of the input channel (IC). Among them, for example, assuming that the number of input channels is 8, then the number of multiplication units inside the PE is equal to 8. Therefore, in the first clock cycle (i.e., time t0), PE0 can process in parallel Figure 12 The L groups of feature data marked with different types of slashes are multiplied and accumulated with the weight data, and there is accumulation calculation within each accumulation tree, but there is no accumulation calculation between accumulation trees.

[0092] At time t1, see Figure 13 , which shows a schematic diagram of the hardware data scheduling sequence of the second clock cycle (time t1) provided by an embodiment of the present application. Figure 13 As shown, the feature data required by the L accumulation trees corresponds to the second pixel in the L sliding windows.

[0093] At time t2, and so on, see Figure 14 , which shows a schematic diagram of the hardware data scheduling sequence of the third clock cycle (t2 moment) provided by an embodiment of the present application. Figure 14 As shown, the feature data required by the L accumulation trees corresponds to the third pixel in the L sliding windows.

[0094] In related art, for a 3×3 sliding window, it takes nine clock cycles to traverse the nine pixels in each sliding window. In other words, for a PE, completing the convolution calculation for L sliding windows requires nine clock cycles. Regardless of the distribution of the weight data, the number of clock cycles required is fixed.

[0095] In the embodiment of the present application, if the pruning strategy in the above embodiment is adopted and non-zero weight data is stored centrally, such as Figure 15 As shown, at this time, each input channel includes 5 non-zero weight data, and the positions of the non-zero weight data in each input channel are: 0 2 4 6 7; 2 4 5 6 7; 0 2 3 4 6; 0 3 4 6 8; then the corresponding feature data can be obtained from the tensor data of the feature image to perform multiplication and accumulation operations. Among them, the non-zero weight data vector filled with left diagonal lines in each input channel participates in the calculation of the first clock cycle, the non-zero weight data vector filled with grid lines participates in the calculation of the second clock cycle, the non-zero weight data vector filled with right diagonal lines participates in the calculation of the third clock cycle, the non-zero weight data vector filled with vertical lines participates in the calculation of the fourth clock cycle, the non-zero weight data vector filled with dots participates in the calculation of the fifth clock cycle, and the remaining weight data filled with white (0 value) does not participate in the calculation. Each time a multiplication and addition operation of a non-zero weight data vector is completed, L accumulation trees will accumulate the L calculation results with the previous calculation result. After 5 clock cycles, the operation that the convolution kernel K0 needs to participate in is completed, and then the compiler can fill in the next batch of data according to the data flow. It can be seen from this that the embodiment of the present application can reduce the calculation time accordingly based on the number of non-zero values of each input channel defined by the algorithm layer; thus, it not only leaves the algorithm layer with sufficient flexibility and exploration space, but also is extremely friendly to hardware, and each multiplication and addition operation can participate in effective data calculation.

[0096] It is understood that the size of the computational array in the embodiment of the present application can be expanded based on actual conditions, and the internal structure of each PE, such as the number of accumulator trees and the number of multiplication units in each accumulator tree, can be scaled based on actual conditions. In addition, the hardware architecture of the embodiment of the present application can not only process 3×3 convolution kernels, but is also applicable to convolution kernels of any other size.

[0097] That is to say, in the embodiments of the present application, first, the technical solution of the present application is aimed at the current pruning strategy and proposes a solution that is more beneficial to hardware implementation. Algorithm pruning and hardware implementation are a joint optimization process. In order to pursue better algorithm effects, a fine-grained pruning method can be used, but the distribution of data is irregular; if a coarse-grained pruning method is used, it can be more friendly to hardware implementation, but the effect loss is large; therefore, the present application proposes a more balanced pruning strategy. While leaving the pruning algorithm with small constraints and a large search space, it tries to consider the friendliness of hardware implementation. By performing balanced pruning on each input channel and adopting a centralized storage strategy for non-zero values, the effective computing power required for algorithm implementation is greatly reduced. In addition, the technical solution of the present application uses a 3×3 convolution kernel as an example, but it can also be applied to convolution kernels of other sizes, such as regular convolution kernels and irregular convolution kernels such as 5×5, 7×7, 1×3, and 3×1, which greatly expands the scope of application; it also makes great use of the hardware resources of multiplication and accumulation operations to achieve better performance and energy efficiency.

[0098] Furthermore, the technical solution of the present application is not only universal, but also reduces the delay of the algorithm implementation through the joint optimization of pruning and architecture, and can achieve the purpose of reducing power consumption by reducing the operating voltage and frequency. For example, assuming that the amount of data to be processed per second is a fixed value, for example, the requirement is to be able to process 30 frames of images per second, that is, the total task volume is the same, then for the general method, each sliding window requires 9 multiplication and accumulation operations to complete, so a higher frequency needs to be set to achieve the goal; and if each sliding window only requires one multiplication and accumulation operation to complete, then in order to achieve the same goal, a lower frequency can be set. Generally speaking, a low frequency means a lower operating voltage requirement, thereby achieving the purpose of reducing power consumption.

[0099] The embodiment of the present application provides a neural network acceleration method for electronic devices. The above embodiment describes the specific implementation of the above embodiment in detail. It can be seen that the technical solution of the above embodiment, on the one hand, can increase the degree of freedom of algorithm pruning, explore effective pruning strategies as much as possible, and reduce the effect loss caused by pruning; on the other hand, through pruning, not only can the algorithm model be compressed, but also the amount of calculation and bandwidth pressure can be reduced through hardware coordination; on the other hand, by jointly optimizing pruning and the zero-jumping technology of the hardware architecture, it can also improve the energy efficiency of the algorithm implementation and reduce the delay of the algorithm implementation while ensuring the algorithm effect.

[0100] In another embodiment of the present application, based on the same inventive concept as the above embodiment, see Figure 16 , which shows a schematic diagram of the structure of a neural network acceleration device 160 provided in an embodiment of the present application. Figure 16 As shown, the neural network acceleration device 160 is applied to an electronic device, and the neural network acceleration device 160 may include: a determination unit 1601, a storage unit 1602 and a calculation unit 1603; wherein,

[0101] The determining unit 1601 is configured to determine a first convolution kernel of the neural network structure; wherein the number of non-zero weight data of each input channel in the first convolution kernel is the same;

[0102] The storage unit 1602 is configured to centrally store the non-zero weight data of each input channel in the first convolution kernel to obtain a second convolution kernel of the neural network structure;

[0103] The computing unit 1603 is configured to perform a convolution operation on the feature image using the second convolution kernel to obtain a target convolution result.

[0104] In some embodiments, see Figure 16 , the neural network acceleration device 160 may further include a pruning unit 1604; wherein,

[0105] The determining unit 1601 is further configured to obtain an initial convolution kernel of the neural network structure;

[0106] The pruning unit 1604 is configured to perform a pruning operation on each input channel in the initial convolution kernel to obtain a first convolution kernel of the neural network structure.

[0107] In some embodiments, the storage unit 1602 is further configured to determine the position index value corresponding to the non-zero weight data of each input channel when the non-zero weight data of each input channel in the first convolution kernel is centrally stored; and store the position index value in a preset storage area of each input channel; wherein the position index value is used to indicate the position of the non-zero weight data in each input channel.

[0108] In some embodiments, the computing unit 1603 is specifically configured to read the position index value of each input channel from the preset storage area of each input channel based on the non-zero weight data of each input channel in the second convolution kernel; and obtain the feature data corresponding to the position index value of each input channel based on the feature image; and multiply the non-zero weight data and the feature data to obtain the multiplication result, and accumulate all the obtained multiplication results to obtain the target convolution result.

[0109] In some embodiments, the electronic device includes a computing array consisting of m×n processing units PE, where m and n are integers greater than zero; wherein each PE includes several accumulator trees, which are used to multiply the received non-zero weight data and the corresponding feature data, and accumulate all multiplication results obtained within the accumulator tree.

[0110] In some embodiments, the computing unit 1603 is further configured to set a number of sliding windows; wherein each sliding window corresponds to an accumulator tree, and each accumulator tree includes C multiplication units, where C represents the number of input channels of the first convolution kernel; and based on the number of sliding windows, the i-th calculation result corresponding to the i-th clock cycle is calculated; and when i is equal to the target clock cycle value, the target convolution result is determined according to the target calculation result corresponding to the target clock cycle value; wherein i is an integer greater than 0.

[0111] In some embodiments, the determination unit 1601 is further configured to determine the number of non-zero weight data for each input channel in the first convolution kernel; and determine the target clock cycle value based on the number of non-zero weight data.

[0112] In some embodiments, the computing unit 1603 is further configured to, when i is equal to 1, determine the first feature data of each sliding window on each input channel in the first clock cycle; utilize the accumulator tree corresponding to each sliding window, multiply the first feature data on each input channel with the first non-zero weight data corresponding to each input channel in the second convolution kernel through C multiplication units, and accumulate the multiplication results corresponding to the C multiplication units through the accumulator tree to obtain a first calculation result.

[0113] In some embodiments, the computing unit 1603 is further configured to determine the i-th feature data of each sliding window on each input channel in the i-th clock cycle when i is greater than 1; utilize the accumulator tree corresponding to each sliding window to multiply the i-th feature data on each input channel with the i-th non-zero weight data corresponding to each input channel in the second convolution kernel through C multiplication units, and accumulate the multiplication results corresponding to the C multiplication units with the i-1-th calculation result obtained last time through the accumulator tree to obtain the i-th calculation result.

[0114] It is understood that in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular system. Furthermore, the various components in this embodiment can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The aforementioned integrated units can be implemented in the form of hardware or software functional modules.

[0115] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0116] Therefore, this embodiment provides a computer storage medium, which stores a computer program. When the computer program is executed by at least one processor, it implements the neural network acceleration method described in any one of the aforementioned embodiments.

[0117] In another embodiment of the present application, based on the composition of the aforementioned neural network acceleration device 160 and the computer storage medium, see Figure 17 , which shows a schematic diagram of the specific hardware structure of an electronic device provided by an embodiment of the present application. Figure 17 As shown, the electronic device 170 may include a processor 1701, which can call and run a computer program from a memory to implement any of the neural network acceleration methods in the aforementioned embodiments.

[0118] Alternatively, as Figure 17 As shown, the electronic device 170 may further include a memory 1702. The processor 1701 may call and run a computer program from the memory 1702 to implement any one of the neural network acceleration methods in the aforementioned embodiments.

[0119] The memory 1702 may be a separate device independent of the processor 1701 , or may be integrated into the processor 1701 .

[0120] Alternatively, as Figure 17 As shown, the electronic device 170 may further include a transceiver 1703 , and the processor 1701 may control the transceiver 1703 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices.

[0121] The transceiver 1703 may include a transmitter and a receiver. The transceiver 1703 may further include an antenna, and the number of antennas may be one or more.

[0122] Optionally, the electronic device 170 may be a device such as a smartphone, tablet computer, PDA, laptop computer, or desktop computer described in the aforementioned embodiments, or a device integrated with the neural network acceleration device 160 described in any of the aforementioned embodiments. Here, the electronic device 170 may include a computing array consisting of m×n processing elements (PEs), where m and n are integers greater than zero; each PE includes a plurality of accumulator trees, which are used to multiply received non-zero weight data with corresponding feature data and accumulate all multiplication results obtained within the accumulator tree; and the electronic device 170 may implement the corresponding processes described in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0123] In another embodiment of the present application, based on the composition of the aforementioned neural network acceleration device 160 and the computer storage medium, in a possible example, see Figure 18 , which shows a schematic diagram of the composition structure of a chip provided in an embodiment of the present application. Figure 18 As shown, the chip 180 may include the neural network acceleration device 160 described in any of the aforementioned embodiments.

[0124] In another possible example, see Figure 19 , which shows a schematic diagram of the specific hardware structure of a chip provided in an embodiment of the present application. Figure 19 As shown, the chip 180 may include a processor 1801, which may call and run a computer program from a memory to implement any of the neural network acceleration methods in the aforementioned embodiments.

[0125] Alternatively, as Figure 19 As shown, the chip 180 may further include a memory 1802. The processor 1801 may call and run a computer program from the memory 1802 to implement any one of the neural network acceleration methods described in the foregoing embodiments.

[0126] The memory 1802 may be a separate device independent of the processor 1801 , or may be integrated into the processor 1801 .

[0127] Alternatively, as Figure 19 As shown, the chip 180 may further include an input interface 1803. The processor 1801 may control the input interface 1803 to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips.

[0128] Alternatively, as Figure 19 As shown, the chip 180 may further include an output interface 1804. The processor 1801 may control the output interface 1804 to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0129] Optionally, the chip 180 can be applied to the electronic device described in the aforementioned embodiment, and the chip can implement the corresponding processes described in the various methods of the embodiments of the present application. For the sake of brevity, they are not described here.

[0130] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc., without any limitation here.

[0131] It should be noted that the processor of the embodiments of the present application may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by hardware integrated logic circuits in the processor or software instructions. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly implemented as a hardware decoding processor, or can be implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0132] It should also be noted that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0133] It is understandable that these embodiments described in the present application can be realized with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. Software code can be stored in memory and executed by processor. Memory can be implemented in processor or outside processor.

[0134] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0135] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0136] It should be noted that, in this application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0137] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0138] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0139] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0140] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0141] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A neural network acceleration method, characterized in that: Applied to electronic equipment, the method includes: Get the initial convolution kernel of the neural network structure; performing a pruning operation on each input channel in the initial convolution kernel to determine a first convolution kernel of the neural network structure; wherein the pruning operation indicates that the number of non-zero weight data of each input channel in the first convolution kernel obtained after pruning is the same, and the neural grid structure is used to determine an output result of at least one of classification, recognition, denoising, and super-resolution corresponding to the input data; Centrally storing the non-zero weight data of each input channel in the first convolution kernel to obtain a second convolution kernel of the neural network structure; Performing a convolution operation on the feature image using the second convolution kernel to obtain a target convolution result; The electronic device includes a computing array, which is composed of m×n processing units PE, where m and n are integers greater than zero; each PE includes a plurality of accumulator trees, which are used to multiply the received non-zero weight data with the corresponding feature data in the feature image, and accumulate all multiplication results obtained within the accumulator tree.

2. The method according to claim 1, characterized in that When centrally storing the non-zero weight data of each input channel in the first convolution kernel, the method further includes: Determine the position index value corresponding to the non-zero weight data of each input channel; The position index value is correspondingly stored in a preset storage area of each input channel; wherein the position index value is used to indicate the position of the non-zero weight data in each input channel.

3. The method according to claim 1, characterized in that The step of performing a convolution operation on the feature image using the second convolution kernel to obtain a target convolution result includes: Based on the non-zero weight data of each input channel in the second convolution kernel, read the position index value of each input channel from the preset storage area of each input channel respectively; Based on the feature image, acquiring feature data corresponding to the position index value of each input channel; The non-zero weight data and the feature data are multiplied to obtain a multiplication result, and all the obtained multiplication results are accumulated to obtain the target convolution result.

4. The method according to claim 1, wherein The step of performing a convolution operation on the feature image using the second convolution kernel to obtain a target convolution result includes: Setting a plurality of sliding windows; wherein each sliding window corresponds to an accumulation tree, and each accumulation tree includes C multiplication units, where C represents the number of input channels of the first convolution kernel; Calculate an i-th calculation result corresponding to an i-th clock cycle based on the plurality of sliding windows; When i is equal to the target clock cycle value, the target convolution result is determined according to the target calculation result corresponding to the target clock cycle value; wherein i is an integer greater than 0.

5. The method according to claim 4, characterized in that After determining the first convolution kernel of the neural network structure, the method further includes: Determining the number of non-zero weight data for each input channel in the first convolution kernel; The target clock cycle value is determined according to the number of non-zero weight data.

6. The method according to claim 4, characterized in that When i is equal to 1, calculating the i-th calculation result corresponding to the i-th clock cycle based on the plurality of sliding windows includes: In the first clock cycle, the first feature data of each sliding window on each input channel is determined; using the accumulator tree corresponding to each sliding window, the first feature data on each input channel is multiplied by the first non-zero weight data corresponding to each input channel in the second convolution kernel through the C multiplication units, and the multiplication results corresponding to the C multiplication units are accumulated through the accumulator tree to obtain a first calculation result.

7. The method according to claim 6, characterized in that When i is greater than 1, calculating the i-th calculation result corresponding to the i-th clock cycle based on the plurality of sliding windows includes: In the i-th clock cycle, determine the i-th feature data of each sliding window on each input channel; use the accumulator tree corresponding to each sliding window to multiply the i-th feature data on each input channel with the i-th non-zero weight data corresponding to each input channel in the second convolution kernel through the C multiplication units, and accumulate the multiplication results corresponding to the C multiplication units and the i-1-th calculation result obtained last time through the accumulator tree to obtain the i-th calculation result.

8. A neural network acceleration device, characterized in that: Applied to electronic equipment, the neural network acceleration device includes a determination unit, a pruning unit, a storage unit and a calculation unit; wherein, The determining unit is configured to obtain an initial convolution kernel of the neural network structure; The pruning unit is configured to perform a pruning operation on each input channel in the initial convolution kernel to determine a first convolution kernel of the neural network structure; wherein the pruning operation indicates that the number of non-zero weight data of each input channel in the first convolution kernel obtained after pruning is the same, and the neural grid structure is used to determine an output result of at least one of classification, recognition, denoising, and super-resolution corresponding to the input data; The storage unit is configured to centrally store the non-zero weight data of each input channel in the first convolution kernel to obtain a second convolution kernel of the neural network structure; The computing unit is configured to perform a convolution operation on the feature image using the second convolution kernel to obtain a target convolution result; The electronic device includes a computing array, which is composed of m×n processing units PE, where m and n are integers greater than zero; each PE includes a plurality of accumulator trees, which are used to multiply the received non-zero weight data with the corresponding feature data in the feature image, and accumulate all multiplication results obtained within the accumulator tree.

9. A chip, characterized in that: The chip includes the neural network acceleration device as described in claim 8.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor; wherein, The memory is used to store a computer program that can be run on the processor; The processor is configured to execute the method according to any one of claims 1 to 7 when running the computer program.

11. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images