Sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation

Abstract

The present invention discloses a sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation, comprising: inputting feature map data of each input channel row by row into a buffer of a data reuse module in sequence, a dynamic non-zero value index module acquiring one row of output feature map data of each input channel and K-1 rows of feature map data gradually output after being cached by each buffer, as well as weight data of each channel, where K is the order of the convolution kernel, and outputting a dynamic non-zero value index after completing data splicing and index calculation; a computing power dynamic allocation module allocating non-zero data pairs in the stored spliced ​​data according to the dynamic non-zero value index; a multiplication-accumulation module performing multiplication-accumulation on the output of the computing power dynamic allocation module, and adjusting the number of multiplication units according to the number of channels, the convolution kernel size and the sparsity based on the optimal utilization rate; the present invention solves the problems of non-zero element pairing and computing resource utilization efficiency of a sparse CNN model through a dynamic non-zero value index and a computing power dynamic allocation mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network accelerators, and more specifically to a sparse convolution acceleration method based on dynamic feature weight dual indexing and computing power allocation. Background Art

[0002] At present, in order to improve the feature extraction capability of convolutional neural networks (CNN) when dealing with complex tasks, the structural complexity and parameter number of CNN models have increased dramatically, and the computing and storage requirements for model operation have also increased accordingly, posing severe challenges to hardware implementation.

[0003] To reduce model complexity and computational complexity, the industry has introduced pruning technology, which reduces complexity by reducing redundant connections while maintaining model accuracy. However, although pruning can reduce network density, it also introduces sparsity problems, resulting in a large number of zero-value calculations, which in turn affects computational efficiency. In addition, the activation function in CNN may exacerbate the sparse distribution of data, further hindering the improvement of computational efficiency. In the multiplication and accumulation process of convolution, these zero-value multiplication calculations do not affect the results and can therefore be regarded as invalid calculations. Convolutional neural networks that contain a large number of zero-value calculations are usually called sparse convolutional neural networks. In recent years, acceleration technology for sparse convolutional neural networks has become one of the research hotspots.

[0004] Patent CN116911357A proposes a convolution calculation accelerator and acceleration method based on CSR encoding. In this invention, before the convolution calculation begins, it is necessary to perform index calculation operations on the blocked feature map, which adds additional calculation time, and it is necessary to record the sparse positions of the blocked feature map, which also brings higher storage overhead; in addition, the invention only indexes the zero value of the feature map, and does not consider the zero value problem of the weight; the use of the multiplication pulsation calculation array will also calculate multiplications containing zero values, and fails to make the multiplier only operate non-zero value multiplication, resulting in a large number of invalid operations.

[0005] Patent CN113705794B proposes a neural network accelerator design method based on dynamic activation bit sparsity. Although this invention realizes dynamic tag activation and reduces index calculation time, it only performs dynamic real-time monitoring and marking of the zero value of the activation data, without fully considering the marking of the zero value of the weight, so that some multipliers still perform invalid operations.

[0006] Patent CN118569329A proposes a hybrid granularity sparse neural network training accelerator and computational acceleration method. This invention can calculate indexes in real time during calculations, but it is necessary to store row bitmaps and element bitmaps of input data and weight data at the same time. The indexing method is complex and occupies a large amount of memory. In addition, the position of non-zero values ​​needs to be further determined during the calculation, resulting in storage and computational overhead.

[0007] Although these studies have made some progress in accelerating sparse convolution calculations, there are still two problems: first, the calculation time of non-zero indexes is long and the index memory usage is too high; second, there are still many zero-value calculations, resulting in insufficient utilization of some multiplication units (MUs).

[0008] Therefore, how to solve the problems of long indexing time and insufficient utilization of multiplication units in the convolution calculation of existing sparse neural network models is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0009] In view of this, the present invention provides a sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation to solve the technical problems mentioned in the background technology.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] A sparse convolution acceleration method based on dynamic feature weight dual indexing and computing power allocation, including a row data multiplexing module, a dynamic non-zero value indexing module, a computing power dynamic allocation module, and a multiplication and accumulation module;

[0012] The specific steps include:

[0013] S1. Sequentially input the feature map data of each input channel line by line into the buffer FIFO of the data multiplexing module, and gradually output the K-1 lines of feature map data after each FIFO cache;

[0014] S2. The dynamic non-zero value index module obtains 1 row of data of each input channel feature map and K-1 rows of feature map data gradually output after each FIFO buffer and the weight data of each channel, completes splicing and index calculation, and outputs the dynamic non-zero value index to the computing power dynamic allocation module;

[0015] S3. The computing power dynamic allocation module allocates non-zero data pairs in the spliced ​​data stored in the data splicing module according to the dynamic non-zero value index output by the dynamic non-zero value index module, and outputs the allocation result to the multiplication and accumulation module;

[0016] S4. The multiplication and accumulation module performs multiplication and accumulation on the output of the dynamic computing power allocation module, and adjusts the number of multiplication units according to the number of channels, convolution kernel size and sparsity to achieve optimal utilization of the multiplication units.

[0017] Preferably, the number of buffer FIFOs is the order K of the convolution kernel minus 1, the feature map of each input channel is represented as an input feature map matrix, and the depth of each FIFO is set to the number of columns of the input feature map matrix.

[0018] Preferably, the dynamic non-zero value index module includes a data splicing module and an index calculation module;

[0019] Data splicing module, which combines the features of each input channel Figure 1 The row data and the K-1 row feature map data gradually output after each FIFO buffer are stored in register A. At the same time, the weight data of each channel is stored in register B. The data in register A is the local area data of the feature map corresponding to the convolution kernel; the data in register B is the convolution kernel data; the convolution kernel data of each channel and the local area data of the feature map corresponding to each channel are spliced ​​separately;

[0020] The index calculation module performs mask indexing on the non-zero values ​​in the input feature map data and weight data of registers A and B in the data splicing module, and then performs position calculation to obtain a dynamic non-zero value index.

[0021] Preferably, the convolution kernel data of each channel and the local area data of the feature map corresponding to each channel are spliced ​​as follows:

[0022] Concatenate the flattened vectors of the C channels in sequence: the first row of data of the first convolution kernel is concatenated with the first row of data of the second convolution kernel, the first row of data of the second convolution kernel is concatenated with the first row of data of the third convolution kernel, and so on until the first row of data of the Cth convolution kernel is concatenated.

[0023] The second row of data of the first convolution kernel is spliced ​​with the second row of data of the second convolution kernel, the second row of data of the second convolution kernel is spliced ​​with the second row of data of the third convolution kernel, and so on until the second row of data of the Kth convolution kernel is spliced ​​together;

[0024] Until the K-th row data of the first convolution kernel is spliced ​​with the second row data of the second convolution kernel, the K-th row data of the second convolution kernel is spliced ​​with the K-th row data of the third convolution kernel, and so on until it is spliced ​​to the K-th row data of the C-th convolution kernel, a two-dimensional matrix of size K×KC is obtained;

[0025] The weight data is stored in the register in the form of a one-dimensional array. The first row of the K×KC two-dimensional matrix is ​​stored in the array at positions 0-(KC-1), the second row is stored in the array at positions KC-2KC, and so on until the Kth row is stored in the array at positions (K-1)KC-(KKC-1).

[0026] The local area data of the feature maps corresponding to each channel are spliced ​​in the same way.

[0027] Preferably, the specific content of the index calculation module obtaining the dynamic non-zero value index is:

[0028] Use the Mask index to mark the inner product multiplication of the input feature map data and weight data; if one of the input feature map data and weight data is zero in the multiplication operation, the multiplication is considered an invalid operation and the corresponding index is set to "0"; if both the input feature map data and weight data are non-zero values, it is considered a valid operation and the corresponding index is set to "1";

[0029] The dynamic non-zero value index is set to a vector, where the 0th bit is the number of non-zero data pairs, and the other bits are the positions of the non-zero data pairs in sequence; all the values ​​in the mask index are summed to get the number of data pairs that need to be multiplied, and stored in the 0th bit of the dynamic non-zero value index array;

[0030] Read the values ​​of the Mask index one by one and record the position in the vector, and determine whether the value is 1. If it is 1, the position information is stored in the dynamic non-zero value index in sequence, otherwise the position information is not stored in the dynamic non-zero value index.

[0031] Preferably, the specific content of step S3 is:

[0032] S31. Divide the 0th bit of the dynamic non-zero index by the number of multiplication units to obtain the quotient q and the remainder r, which correspond to the number of cycles when the multiplication unit is fully loaded and the number of multiplication units required to be calculated when not fully loaded;

[0033] S32. Record the calculation progress by cycle count t. When an inner product operation starts, count from 0 and record the number of cycles spent on the current inner product operation.

[0034] S33 obtains the input feature map value and weight data at different cycles to calculate the position base, which is used to determine the address of the data in register A and register B at different cycles;

[0035] S34. Determine whether the current calculation state is within the valid range of the full load cycle or the non-full load cycle, and obtain the number of multiplication units performing the calculation;

[0036] S35 obtains the feature map data and the weight data calculated by the current multiplication unit, and calculates the position information of each cycle of the inner product operation;

[0037] S36. Obtain the non-zero data in the corresponding multiplication positions of the input feature map data in register A and the weight data in register B according to the position information, and output them to the multiplication and accumulation module.

[0038] Preferably, in step S4, when the computing unit is fully loaded, D is accumulated. MUThe product of the multiplication units. When the computing unit is not fully loaded, accumulate the products of r multiplication units. Start accumulating from the first cycle of the inner product calculation, and add the calculation results of each cycle until the inner product calculation ends to obtain a value of the output feature map.

[0039] Preferably, in step S32, when t is less than q, it indicates that the current inner product operation calculation is not completed, and t is incremented by 1 while each of the D MU multiplication units completes one calculation per cycle; when t is equal to q, it indicates that one inner product operation calculation ends, t is cleared, and the number of cycles spent on the next inner product operation is started to be recorded;

[0040] In step S33, the position base Pos is: Pos = t × D MU ;

[0041] In step S34, when t < q, all multiplication units are in the fully loaded state, and at this time, the no-load problem does not need to be considered; when t == q && MU_num <= r, some multiplication units are in the idle state, and only r multiplication units are used to perform the calculation. MU_num is the multiplication unit number, and its value range is 1 - D MU 。

[0042] Preferably, the specific content of step S35 is:

[0043] Let the currently obtained input feature map data be X, and let the currently obtained weight data be W. X = IFM[DNV[MU_num + Pos]], W = weight[DNV[MU_num + Pos]];

[0044] In the 0th cycle of the inner product operation, t = 0, Pos is 0, and the position information of DNV[1] - DNV[D MU will be taken out. In the 1st cycle of the inner product operation, t = 1, Pos is D MU , and the position information of DNV[1 + D MU - DNV[D MU + D MU will be taken out. In the 2nd cycle of the inner product operation, t = 2, Pos is 2 × D MU , and the position information of DNV[1 + 2 × D MU - DNV[D MU + 2 × D MU will be taken out, and so on;

[0045] In the last cycle of the inner product operation, when some multiplication units do not meet the conditions of t < q || (t == q && MU_num <= r), only the position information that meets the operation conditions is taken out, and the multiplication units that do not meet the conditions are made idle.

[0046] Preferably, the multiplication unit utilization rate when the convolution kernel performs the inner product operation calculation with the corresponding local area of ​​the feature map is:

[0047]

[0048] Among them, S local is the sparsity of the inner product matrix, K is the order of the convolution kernel, C is the number of channels, and D MU is the number of multiplication units, The symbol for rounding up.

[0049] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a sparse convolution acceleration method based on dynamic feature weight dual indexing and computing power allocation. Through dynamic non-zero value indexing, features and weights can be indexed at the same time, and the calculation time and memory requirements of the index are reduced. A dynamic computing power allocation strategy is adopted to perform computing power allocation operations on data after zero jump, and the data after zero jump of multiple channel convolutions are concentratedly allocated to a group of multiplication units, which reduces the difficulty of pairing non-zero data. The number of multiplication units can be adjusted as needed according to the number of channels, convolution kernel size and sparsity to achieve optimal utilization of multiplication units and avoid idle computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0051] Figure 1 A schematic diagram of a sparse convolution acceleration method based on dynamic feature weight dual indexing and computing power allocation provided by the present invention;

[0052] Figure 2 A schematic diagram of the data flow of the buffer FIFO provided by the present invention;

[0053] Figure 3 Schematic diagram of the process of updating stored data in the data reuse module provided by the present invention;

[0054] Figure 4 Schematic diagram of the splicing operation of convolution kernel data and feature map local area data provided by the present invention;

[0055] Figure 5 Schematic diagram of index calculation provided by the present invention;

[0056] Figure 6 This is a logic diagram of the circuit implementation of the computing power dynamic allocation module provided by the present invention;

[0057] Figure 7 This is a schematic diagram of allocating data through indexes provided by the present invention. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0059] The embodiment of the present invention discloses a sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation, such as Figure 1 , including row data multiplexing module, dynamic non-zero value index module, computing power dynamic allocation module and multiplication and accumulation module;

[0060] The specific steps include:

[0061] S1. Sequentially input the feature map data of each input channel line by line into the buffer FIFO of the data multiplexing module, and gradually output the K-1 lines of feature map data after each FIFO cache;

[0062] S2. The dynamic non-zero value index module obtains 1 row of data of each input channel feature map and K-1 rows of feature map data gradually output after each FIFO buffer and the weight data of each channel, completes splicing and index calculation, and outputs the dynamic non-zero value index to the computing power dynamic allocation module;

[0063] S3. The computing power dynamic allocation module allocates non-zero data pairs in the spliced ​​data stored in the data splicing module according to the dynamic non-zero value index output by the dynamic non-zero value index module, and outputs the allocation result to the multiplication and accumulation module;

[0064] S4. The multiplication and accumulation module performs multiplication and accumulation on the output of the dynamic computing power allocation module, and adjusts the number of multiplication units according to the number of channels, convolution kernel size and sparsity to achieve optimal utilization of the multiplication units.

[0065] In order to further implement the above technical solution, the number of buffer FIFOs is the order K of the convolution kernel minus 1, the feature map of each input channel is represented as an input feature map matrix, and the depth of each FIFO is set to the number of columns of the input feature map matrix.

[0066] In this embodiment, if Figure 2The buffer FIFO is first-in-first-out, and its working method is as follows: starting from the 1st channel to the Cth channel, the input feature map matrix of each channel is input into the FIFO row by row; starting from the first row, each row of data of the input feature map matrix is ​​input into the 1st FIFO in sequence, and is also output to the dynamic non-zero value index module; when the 1st FIFO is full, its output data is input into the 2nd FIFO in sequence, and is also output to the dynamic non-zero value index module; when the 2nd FIFO is full, the data therein is input into the 3rd FIFO in sequence, and is also output to the dynamic non-zero value index module, and so on.

[0067] In order to further implement the above technical solution, the dynamic non-zero value index module includes a data splicing module and an index calculation module;

[0068] Data splicing module, such as Figure 3 , each input channel feature Figure 1 The row data and the K-1 row feature map data gradually output after each FIFO buffer are stored in register A. At the same time, the weight data of each channel is stored in register B. The data in register A is the local area data of the feature map corresponding to the convolution kernel; the data in register B is the convolution kernel data; the convolution kernel data of each channel and the local area data of the feature map corresponding to each channel are spliced ​​separately;

[0069] In this embodiment, for one row of data of the external input channel feature map, each channel stores only one data in each cycle. After K data are stored, the data input in the current cycle replaces the data in the previous K cycle. The C input channels store a total of C×K data, which are stored in register A. Each FIFO output stores only one data in each cycle. After K data are stored, the data input in the current cycle replaces the data in the previous K cycle. The C×(K-1) FIFO outputs store a total of C×(K-1)×K data, which are stored in register A.

[0070] The weight data of the C channels of external input, each channel only stores 1 data in each cycle, and the C input channels store a total of C×K×K data, which are stored in register B;

[0071] The index calculation module performs mask indexing on the non-zero values ​​in the input feature map data and weight data of registers A and B in the data splicing module, and then performs position calculation to obtain the dynamic non-zero value DNV index.

[0072] In order to further implement the above technical solutions, Figure 4 , the convolution kernel data of each channel and the local area data of the feature map corresponding to each channel are spliced ​​as follows:

[0073] Concatenate the flattened vectors of the C channels in sequence: the first row of data of the first convolution kernel is concatenated with the first row of data of the second convolution kernel, the first row of data of the second convolution kernel is concatenated with the first row of data of the third convolution kernel, and so on until the first row of data of the Cth convolution kernel is concatenated.

[0074] The second row of data of the first convolution kernel is spliced ​​with the second row of data of the second convolution kernel, the second row of data of the second convolution kernel is spliced ​​with the second row of data of the third convolution kernel, and so on until the second row of data of the Kth convolution kernel is spliced ​​together;

[0075] Until the K-th row data of the first convolution kernel is spliced ​​with the second row data of the second convolution kernel, the K-th row data of the second convolution kernel is spliced ​​with the K-th row data of the third convolution kernel, and so on until it is spliced ​​to the K-th row data of the C-th convolution kernel, a two-dimensional matrix of size K×KC is obtained;

[0076] The weight data is stored in the register in the form of a one-dimensional array. The first row of the K×KC two-dimensional matrix is ​​stored in the array at positions 0-(KC-1), the second row is stored in the array at positions KC-2KC, and so on until the Kth row is stored in the array at positions (K-1)KC-(KKC-1).

[0077] The local area data of the feature maps corresponding to each channel are spliced ​​in the same way.

[0078] In order to further implement the above technical solutions, Figure 5 , the specific content of the index calculation module to obtain the dynamic non-zero value index is:

[0079] Use the Mask index to mark the inner product multiplication of the input feature map data and weight data; if one of the input feature map data and weight data is zero in the multiplication operation, the multiplication is considered an invalid operation and the corresponding index is set to "0"; if both the input feature map data and weight data are non-zero values, it is considered a valid operation and the corresponding index is set to "1";

[0080] In this embodiment, a total of n pairs of data are judged, and the input feature map data and weight data that need to be multiplied are a pair, and n is the number of elements in the concatenated K×KC matrix;

[0081] The dynamic non-zero value index is set as a vector, where the 0th bit is the number of non-zero data pairs, and the other bits are the positions of non-zero data pairs in sequence; sum all the values ​​in the Mask index to get the data pairs N that need to be multiplied nz The number of and stored in the 0th position of the array of dynamic non-zero index;

[0082] Read the values ​​of the Mask index one by one and record the position in the vector, and determine whether the value is 1. If it is 1, the position information is stored in the dynamic non-zero value index in sequence, otherwise the position information is not stored in the dynamic non-zero value index.

[0083] In order to further implement the above technical solutions, Figure 6 , the specific content of step S3 is:

[0084] S31. Divide the 0th bit data by the number D of multiplication units MU using the dynamic non-zero value index MU The quotient q and remainder r are obtained, which correspond to the number of cycles when the multiplication unit is fully loaded and the number of multiplication units that need to be calculated when not fully loaded;

[0085] S32. Record the calculation progress by cycle count t. When an inner product operation starts, count from 0 and record the number of cycles spent on the current inner product operation.

[0086] S33 obtains the input feature map value and weight data at different cycles to calculate the position base Pos, which is used to determine the address of the data in registers A and B at different cycles;

[0087] S34. Determine whether the current calculation state is within the valid range of the full load cycle or the non-full load cycle, and obtain the multiplication unit that performs the calculation;

[0088] S35 obtains the feature map data and the weight data calculated by the current multiplication unit, and extracts the position information of each cycle of the inner product operation;

[0089] S36. Obtain the non-zero data in the corresponding multiplication positions of the input feature map data in register A and the weight data in register B according to the position information, and output them to the multiplication and accumulation module.

[0090] In order to further implement the above technical solution, in step S4, when the computing unit is fully loaded, D MU When the computing unit is not fully loaded, the products of r multiplication units are accumulated, starting from the first cycle of the inner product calculation. The calculation results of each cycle are added until the inner product calculation is completed to obtain a value of the output feature map.

[0091] In order to further implement the above technical solution, in step S32, when t is less than q, it means that the current inner product operation calculation is not completed, D MU Each multiplication unit completes a calculation in each cycle and t is incremented by 1. When t is equal to q, it means that the calculation of an inner product operation is completed, t is reset to zero, and the number of cycles spent on the next inner product operation begins to be recorded.

[0092] In step S33, the position cardinality Pos is: Pos = t × D MU ;

[0093] In step S34, when t < q, all multiplication units are in the full-load state, and there is no need to consider the no-load problem at this time; when t == q && MU_num <= r, some multiplication units are in the idle state, and only r multiplication units are used to perform calculations. MU_num is the multiplication unit number, and its value range is 1 - D MU .

[0094] To further implement the above technical solution, the specific content of step S35 is as follows:

[0095] Let the currently obtained input feature map data be X, and let the currently obtained weight data be W. X = IFM[DNV[MU_num + Pos]], W = weight[DNV[MU_num + Pos]];

[0096] In the 0th cycle of the inner product operation, t = 0, Pos is 0, and the position information of DNV[1] - DNV[D MU will be retrieved. In the 1st cycle of the inner product operation, t = 1, Pos is D MU , and the position information of DNV[1 + D MU - DNV[D MU + D MU will be retrieved. In the 2nd cycle of the inner product operation, t = 2, Pos is 2 × D<00,00027>, and the position information of DNV[1 + 2 × D MU - DNV[D MU + 2 × D MU will be retrieved, and so on;

[0097] In the last cycle of the inner product operation, when some multiplication units do not meet the conditions of t < q || (t == q && MU_num <= r), only the position information that meets the operation conditions is retrieved, and the multiplication units that do not meet the conditions are made idle.

[0098] To further implement the above technical solution, the number N of non-zero value data pairs in the process of performing an inner product operation between the convolution kernel and the corresponding local region of the feature map nz is:

[0099] N nz = S local K 2 C

[0100] where S local represents the proportion of non-zero values in C local regions of the feature map when the convolution kernel slides to a specific position in the convolution operation. K is the convolution kernel order, and C is the channel size;

[0101] The number of cycles T in the process of performing an inner product operation between the convolution kernel and the corresponding local region of the feature map total is:

[0102]

[0103] Among them, D MU represents the number of multiplication units, is the round-up symbol, when If it is not an integer, the calculation cycle needs to be rounded up. This means that in the last cycle, even if the multiplication unit is not fully loaded, a cycle of calculation must be performed. At this time, the multiplication unit will be partially idle.

[0104] The multiplication unit utilization U of the inner product operation calculation process performed by the convolution kernel and the corresponding feature map local area is:

[0105]

[0106] Among them, T total D MU Indicates that all multiplication units are in T total The total number of working times in a cycle, including effective and invalid calculations, N nz Indicates the number of non-zero value data pairs that need to be calculated, and also represents the actual effective calculation times of the multiplication unit;

[0107] Then, when the convolution kernel and the corresponding local area of ​​the feature map perform the inner product operation, the multiplication unit utilization is:

[0108]

[0109] Among them, S local is the sparsity of the inner product matrix, K is the order of the convolution kernel, C is the number of channels, and D MU is the number of multiplication units, is the rounding symbol;

[0110] Since the inner product matrix is ​​sparse S local , the convolution kernel size K and the number of channels C are fixed, so the number of multiplication units D can be changed MU Maximize the multiplication unit utilization U.

[0111] In another embodiment, the size of the input feature map is 28×28, the size of the convolution kernel is 3×3, the input channels are 4, and the number of multiplication units is 8;

[0112] The data multiplexing module uses multiple first-in-first-out buffers (FIFOs) to receive input feature map data from four channels. The FIFO module can be generated using the IP core RAM-based Shift Register to set parameters, or the module can be written according to logic.

[0113] The number of FIFOs is the order K of the convolution kernel minus 1. If the convolution kernel size is 3×3, then 2 FIFOs are required.

[0114] The input feature map of each channel is represented as an input feature map matrix;

[0115] The depth of each FIFO is set to the number of columns of the input feature map matrix. If the size of the input feature map is 28×28, then the FIFO depth is set to 28, and each FIFO stores 28 data;

[0116] Starting from the first channel to the fourth channel, the input feature map matrix of each channel is input into the FIFO row by row; starting from the first row, each row of the input feature map matrix is ​​sequentially input into the first FIFO and output to the dynamic non-zero value index module; when the first FIFO is full, its output data is sequentially input into the second FIFO and output to the dynamic non-zero value index module; when the second FIFO is full, its data is sequentially input into the third FIFO and output to the dynamic non-zero value index module;

[0117] The dynamic non-zero value index module includes a data splicing module and an index calculation module. It receives one line of data from the four input channel feature maps and two lines of feature map data outputted by the data multiplexing module. It also receives the weight data of the four channels inputted from the external input. After completing the index calculation, it outputs the DNV index to the dynamic allocation module.

[0118] The data splicing module receives 4 input channel features from external input Figure 1 2 rows of feature map data output by the row data and data multiplexing module; 1 row of data of the 4 input channel feature maps input externally, each channel stores only 1 data per cycle, after 3 data are stored, the data input in the current cycle will replace the data of the previous 3 cycles, 4 input channels store a total of 12 data, stored in register A, each FIFO output stores only 1 data per cycle, after 3 data are stored, the data input in the current cycle will replace the data of the previous 3 cycles, 8 FIFO outputs store a total of 24 data, stored in register A, register A stores a total of 36 data, the data in register A is the feature map local area data corresponding to the convolution kernel;

[0119] Receive the input channel feature map data and the weight data of the four channels of external input at the same time. Each channel only stores one data in each cycle. The four input channels store a total of 36 data in register B. The data in register B is the convolution kernel data.

[0120] Register A and register B can be implemented using a lookup table;

[0121] The convolution kernel data of the four channels and the local area data of the feature maps corresponding to the four channels are spliced ​​respectively; the flattened vectors of the four channels are spliced ​​in sequence, the first row of data of the first convolution kernel is spliced ​​with the first row of data of the second convolution kernel, the first row of data of the second convolution kernel is spliced ​​with the first row of data of the third convolution kernel, the first row of data of the third convolution kernel is spliced ​​with the first row of data of the fourth convolution kernel; the second row of data of the first convolution kernel is spliced ​​with the second convolution kernel The second row of data, the second row of data of the second convolution kernel is spliced ​​with the second row of data of the third convolution kernel, the second row of data of the third convolution kernel is spliced ​​with the second row of data of the fourth convolution kernel; the third row of data of the first convolution kernel is spliced ​​with the third row of data of the second convolution kernel, the third row of data of the second convolution kernel is spliced ​​with the third row of data of the third convolution kernel, and the third row of data of the third convolution kernel is spliced ​​with the third row of data of the fourth convolution kernel; a two-dimensional matrix of size 3×12 is obtained;

[0122] The weight data is stored in the register in the form of a one-dimensional array. The first row of the 3×12 two-dimensional matrix is ​​stored in the array at positions 0-11, the second row is stored in the array at positions 12-23, and so on until the third row is stored in the array at positions 24-35.

[0123] Similarly, the corresponding local areas of the feature maps are spliced;

[0124] The index calculation module performs mask indexing on the non-zero values ​​in the input feature map data and weight data of registers A and B in the data splicing module, and then inputs them into the position calculation module to obtain a dynamic non-zero value index;

[0125] The index calculation module first uses the mask index to mark the inner product operation multiplication; a total of 36 pairs of data are judged, and the input feature map data and weight data that need to be multiplied are a pair; if one of the input feature map data and weight data is zero in the multiplication operation, the multiplication is considered an invalid operation and the corresponding index is set to "0"; if both the input feature map data and weight data are non-zero values, it is considered a valid operation and the corresponding index is set to "1";

[0126] The dynamic non-zero value index is set to a vector, where the 0th bit is the number of non-zero data pairs, and the other bits are the positions of the non-zero data pairs in sequence;

[0127] Sum all the values ​​in the Mask index to get the data pair N that needs to be multiplied. nz The number of and store it in the 0th position of the array of dynamic non-zero index;

[0128] The position calculation module reads the value of the Mask index one by one and records the position in the vector, and then determines whether its value is 1. If it is 1, the position information is stored in the dynamic non-zero value index in sequence. If it is not 1, the position information is not stored in the dynamic non-zero value index.

[0129] The computing power dynamic allocation module allocates non - zero data pairs among the concatenated data stored in register A and register B in the data concatenation module according to the dynamic non - zero value index output by the dynamic non - zero value indexing module, and outputs the allocation result to the multiply - accumulate module;

[0130] The quotient q and remainder r obtained by dividing the 0 - th bit data of the dynamic non - zero value by the number of multiplication units 8 respectively correspond to the number of cycles when the multiplication units are fully loaded and the number of multiplication units that need to perform calculations when not fully loaded;

[0131] In this embodiment, the divider can use a lookup table to accelerate division, that is, all quotients and remainders are pre - calculated and stored in the lookup table, and the lookup table returns the quotient and remainder through direct indexing. This method can save resources better than a general divider and avoid timing problems.

[0132] Specifically: the value of the dynamic non - zero value is 14, calculate Get is the floor function symbol. It is calculated that there is 1 fully - loaded cycle, 14 % 8 = 6 to get r, and it is calculated that 6 multiplication units are used in the non - fully - loaded cycle.

[0133] The cycle count t records the calculation progress; at the start of calculating an inner - product operation, it starts counting from 0 and records the number of cycles that the current inner - product operation has spent; when t is less than q, it means that the calculation of the current inner - product operation is not completed, and while 8 multiplication units complete one calculation per cycle, t is incremented by 1; when t is equal to q, it means that the calculation of an inner - product operation ends, t is cleared, and the number of cycles that the next inner - product operation has spent starts to be recorded;

[0134] Then, a position base Pos is calculated for the input feature map values and weight data at different cycles, Pos = t×8, which is used to determine the addresses of the data in register A and register B at different cycles;

[0135] Judge whether it is within the valid range of the fully - loaded cycle or non - fully - loaded cycle. When t < q, all multiplication units are in the fully - loaded state, and at this time, there is no need to consider the idle problem; the 0 - th cycle is in the fully - loaded state, and non - zero data pairs are allocated according to the number of bits of the array. The 1st to 8th bits of the array are allocated to the 1st to 8th multiplication units for the calculation of the 1st cycle, and 8 groups of non - zero data pairs are input; when (t == q && MU_num <= r), some multiplication units are in the idle state, and only r multiplication units are used to perform calculations; the 1st cycle is in the non - fully - loaded state. For the 9th to 14th bits, 1 to 6 calculated by taking the remainder are allocated to the 1st to 6th multiplication units in the 2nd cycle, and 6 groups of non - zero data pairs are input. MU_num is the multiplication unit number, and its value range is 1 - 8. The result is as Figure 7 shown;

[0136] After determining whether the current calculation state is in a full-load cycle or a non-full-load cycle, obtain the data calculated by the current multiplication unit. Let the data of the current input feature map be X and the data of the current weight be W.

[0137] X = IFM[DNV[MU_num + 8]], W = weight[DNV[MU_num + 8]]; At the 0th cycle of the inner product operation, t = 0, Pos is 0, and the position information of DNV[1]-DNV[8] will be taken out. At the 1st cycle of the inner product operation, t = 1, Pos is 8. There are cases where the 7th and 8th multiplication units do not meet the condition of t < q || (t == q && MU_num <= r). The position information of DNV[9]-DNV

[14] will be taken out, and the position information of DNV

[15] -DNV

[16] will not be taken out, and the 7th and 8th multiplication units will be idle.

[0138] Obtain the data that is non-zero at the corresponding multiplication positions of the input feature map data in register A and the weight data in register B according to the position information, and output it to the multiply-accumulate module.

[0139] The multiply-accumulate module performs multiply-accumulation on the output of the computing power dynamic allocation module. When the computing unit is in a full-load state, the products of 8 multiplication units are accumulated. When the computing unit is in a non-full-load state, the products of r multiplication units are accumulated. Starting from the first cycle of the inner product calculation, the calculation results of each cycle are added together until the inner product calculation ends, and a value of the output feature map is obtained; when performing an inner product operation on the convolution kernel and the corresponding local area of the feature map, the inner product results of multiple input channels of the same output channel can be directly added; therefore, during the calculation process, the results can be gradually accumulated; in this example, there are 14 multiplication results for 4 input channels. At the 0th cycle, the sum of eight numbers is calculated. At the 1st cycle, the result of the 0th cycle is accumulated, and then 6 more numbers are added; the value of one inner product is calculated, that is, a value of the output feature map.

[0140] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0141] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation, characterized in that: It includes row data multiplexing module, dynamic non-zero value index module, computing power dynamic allocation module and multiplication and accumulation module; The specific steps include: S1. Sequentially input the feature map data of each input channel line by line into the buffer FIFO of the data multiplexing module, and gradually output the K-1 lines of feature map data after each FIFO cache; S2. The dynamic non-zero value index module obtains 1 row of data from each input channel feature map and K-1 rows of feature map data gradually output after being buffered by each FIFO, as well as the weight data of each channel, where K is the order of the convolution kernel. After completing splicing and index calculation, it outputs the dynamic non-zero value index DNV to the dynamic computing power allocation module; S3. The computing power dynamic allocation module allocates non-zero data pairs in the spliced ​​data stored in the data splicing module according to the dynamic non-zero value index output by the dynamic non-zero value index module, and outputs the allocation result to the multiplication and accumulation module; S4. The multiplication-accumulation module performs multiplication and accumulation on the output of the dynamic computing power allocation module and adjusts the number of multiplication units based on the number of channels, convolution kernel size, and sparsity to achieve optimal utilization of the multiplication units. The number of buffer FIFOs is the order K of the convolution kernel minus 1. The feature map of each input channel is represented as an input feature map matrix, and the depth of each FIFO is set to the number of columns of the input feature map matrix. The dynamic non-zero value index module includes a data splicing module and an index calculation module; The data splicing module stores 1 row of feature map data of each input channel and K-1 rows of feature map data gradually output after being buffered by each FIFO into register A. At the same time, the weight data of each channel is stored into register B. The data in register A is the local area data of the feature map corresponding to the convolution kernel; the data in register B is the convolution kernel data; the convolution kernel data of each channel and the local area data of the feature map corresponding to each channel are spliced ​​separately; The index calculation module performs mask indexing on the non-zero values ​​in the input feature map data and weight data of registers A and B in the data splicing module, and then performs position calculation to obtain a dynamic non-zero value index.

2. The sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation according to claim 1 is characterized in that: The contents of splicing the convolution kernel data of each channel and the local area data of the feature map corresponding to each channel are as follows: Concatenate the flattened vectors of the C channels in sequence: the first row of data of the first convolution kernel is concatenated with the first row of data of the second convolution kernel, the first row of data of the second convolution kernel is concatenated with the first row of data of the third convolution kernel, and so on until the first row of data of the Cth convolution kernel is concatenated. The second row of data of the first convolution kernel is spliced ​​with the second row of data of the second convolution kernel, the second row of data of the second convolution kernel is spliced ​​with the second row of data of the third convolution kernel, and so on until the second row of data of the Kth convolution kernel is spliced ​​together; Until the K-th row data of the first convolution kernel is spliced ​​with the second row data of the second convolution kernel, the K-th row data of the second convolution kernel is spliced ​​with the K-th row data of the third convolution kernel, and so on until it is spliced ​​to the K-th row data of the C-th convolution kernel, a two-dimensional matrix of size K×KC is obtained; The weight data is stored in the register in the form of a one-dimensional array. The first row of the K×KC two-dimensional matrix is stored in the 0-(KC-1) bits of the array, the second row is stored in the KC-2KC bits of the array, and so on until the Kth row is stored in the (K-1)KC-(KKC-1) bits of the array; Similarly, the data of the local regions of the feature maps corresponding to each channel are concatenated.

3. The sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation according to claim 1 is characterized in that: The specific content obtained by the index calculation module for the dynamic non-zero value index is as follows: Use the Mask index to mark the multiplication operation of the inner product of the input feature map data and the weight data; if one of the input feature map data and the weight data in the multiplication operation is zero, the multiplication is regarded as an invalid operation, and the corresponding index is set to "0"; if both the input feature map data and the weight data are non-zero values, it is regarded as a valid operation, and the corresponding index is set to "1"; The dynamic non-zero value index is set to a vector. Its 0th bit is the number of non-zero data pairs, and the other bits are the positions of the non-zero data pairs in sequence; sum all the values in the Mask index to obtain the number of data pairs that need to perform multiplication operations, and store it in the 0th bit of the array of the dynamic non-zero value index; Read the values of the Mask index one by one and record their positions in the vector. Determine whether the value is 1. If it is 1, store the position information in the dynamic non-zero value index in sequence, otherwise do not store the position information in the dynamic non-zero value index.

4. The sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation according to claim 3 is characterized in that: The specific content of step S3 is as follows: S31. Divide the data in the 0th bit of the dynamic non-zero value index by the number of multiplication units to obtain the quotient q and the remainder r, which respectively correspond to the number of cycles when the multiplication units are full and the number of multiplication units that need to perform calculations when they are not full; S32. Record the calculation progress through the cycle count t. When the calculation of an inner product operation starts, count from 0 and record the number of cycles that the current inner product operation has spent; S33. Calculate the position base for the input feature map values and the weight data at different cycles, which is used to determine the addresses of the data in register A and register B at different cycles; S34. Determine whether the current calculation state is within the valid range of the full-load cycle or the non-full-load cycle to obtain the number of multiplication units that perform calculations; S35. Obtain the feature map data calculated by the current multiplication unit and the current weight data, and calculate the position information at each cycle of the inner product operation; S36. According to the position information, obtain the data that is non-zero at the corresponding multiplication positions of the input feature map data in register A and the weight data in register B, and output it to the multiply-accumulate module.

5. The sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation according to claim 4 is characterized in that: In step S4, when the computing unit is fully loaded, D MU When the computing unit is not fully loaded, the products of r multiplication units are accumulated, starting from the first cycle of the inner product calculation. The calculation results of each cycle are added until the inner product calculation is completed to obtain a value of the output feature map.

6. The sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation according to claim 4 is characterized in that: In step S32, when t is less than q, it means that the current inner product operation calculation is not completed, D MU Each multiplication unit completes a calculation in each cycle and t is incremented by 1. When t is equal to q, it means that the calculation of an inner product operation is completed, t is reset to zero, and the number of cycles spent on the next inner product operation begins to be recorded. In step S33, the position cardinality Pos is: Pos = t × D MU ; In step S34, when t < q, all multiplication units are in the full-load state, and there is no need to consider the no-load problem at this time; when t == q && MU_num <= r, some multiplication units are in the idle state, and only r multiplication units are used to perform calculations. MU_num is the multiplication unit number, and its value range is 1 - D MU .

7. The sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation according to claim 4 is characterized in that: The specific content of step S35 is as follows: Let the currently obtained input feature map data be X, and the currently obtained weight data be W. X = IFM[DNV[MU_num+Pos]], W = weight[DNV[MU_num+Pos]]; In the 0th cycle of the inner product operation, t=0, Pos is 0, and DNV[1]-DNV[D MU ] position information, in the first cycle of the inner product operation, t = 1, Pos is D MU , will take out DNV[1+D MU ]-DNV[D MU +D MU ] position information, in the second cycle of the inner product operation, t = 2, Pos is 2 × D MU , will take out DNV[1+2×D MU ]-DNV[D MU +2×D MU ] location information, and so on; At the last cycle of the inner product operation, when some multiplication units do not meet the condition of t<q||(t==q&&MU_num<=r), only the position information that meets the operation condition is taken out, and the multiplication units that do not meet the condition are left idle.

8. The sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation according to claim 1 is characterized in that: When the convolution kernel performs an inner product operation with the corresponding local region of the feature map, the utilization rate of the multiplication units is: Among them, S local is the sparsity of the inner product matrix, K is the order of the convolution kernel, C is the number of channels, and D MU is the number of multiplication units, The symbol for rounding up.

Images