Sparse convolution acceleration method based on dynamic feature weight double indexes and computing power distribution

Abstract

The invention discloses a sparse convolution acceleration method based on dynamic feature weight double indexes and computing power distribution, which comprises the following steps: sequentially inputting feature map data of each input channel into a buffer of a data multiplexing module line by line, the dynamic non-zero index module obtains one line of data of each output input channel feature map, (K-1) lines of feature map data output step by step after being cached by each buffer and weight data of each channel, K is a convolution kernel order, and a dynamic non-zero index is output after data splicing and index calculation are completed; the computing power dynamic allocation module allocates non-zero data pairs in the stored spliced data according to the dynamic non-zero index; the multiply-accumulate module performs multiply-accumulate on the output of the computing power dynamic allocation module, and adjusts the number of multiplication units according to the channel number, the convolution kernel size and the sparseness based on the optimal utilization rate; according to the method, the problems of non-zero element pairing and computing resource utilization efficiency of the sparse CNN model are solved through the dynamic non-zero index and the computing power dynamic allocation mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network accelerators, and more specifically to a sparse convolution acceleration method based on dynamic feature weight dual indexing and computing power allocation. Background Art

[0002] At present, in order to improve the feature extraction capability of convolutional neural networks (CNN) when coping with complex tasks, the structural complexity and parameter quantity of CNN models have increased dramatically, and the computing and storage requirements for model operation have also increased accordingly, posing severe challenges to hardware implementation.

[0003] In order to reduce model complexity and computational complexity, the industry has introduced pruning technology to reduce complexity by reducing redundant connections while maintaining model accuracy. However, although pruning can reduce network density, it also introduces sparsity problems, resulting in a large number of zero-value calculations, which in turn affects computational efficiency. In addition, the activation function in CNN may aggravate the sparse distribution of data, further hindering the improvement of computational efficiency. In the multiplication and accumulation process of convolution, these zero-value multiplication calculations do not affect the results, so they can be regarded as invalid calculations. Convolutional neural networks that contain a large number of zero-value calculations are usually called sparse convolutional neural networks. In recent years, acceleration technology for sparse convolutional neural networks has become one of the research hotspots.

[0004] Patent CN116911357A proposes a convolution calculation accelerator and acceleration method based on CSR coding. In this invention, before the convolution calculation starts, it is necessary to perform index calculation operations on the blocked feature map, which increases the additional calculation time, and it is necessary to record the sparse positions of the blocked feature map, which also brings higher storage overhead; in addition, this invention only indexes the zero value of the feature map, and does not consider the zero value problem of the weight; using the multiplication pulsation calculation array will also calculate multiplications containing zero values, and fails to make the multiplier only calculate non-zero value multiplication, resulting in a large number of invalid operations.

[0005] Patent CN113705794B proposes a neural network accelerator design method based on dynamic activation bit sparsity. Although dynamic label activation is implemented in this invention and the index calculation time is reduced, only the zero value of the activation data is dynamically monitored and marked in real time, and the marking of the zero value of the weight is not fully considered, so that some of the multipliers still perform invalid operations.

[0006] Patent CN118569329A proposes a hybrid granularity sparse neural network training accelerator and a computing acceleration method. The invention can calculate the index in real time during calculation, but it is necessary to store the row bitmap and element bitmap of the input data and weight data at the same time. The indexing method is complex and occupies a large amount of memory. In addition, the non-zero value position needs to be further judged during the calculation, resulting in storage and computing overhead.

[0007] Although these studies have made some progress in accelerating sparse convolution calculations, there are still two problems: first, the non-zero index calculation time is long and the index memory usage is too high; second, it still contains many zero-value calculations, resulting in insufficient utilization of some multiplication units (MU).

[0008] Therefore, how to solve the problems of long indexing time and insufficient utilization of multiplication units in the convolution calculation of the existing sparse neural network model is an urgent problem that technicians in this field need to solve. Summary of the invention

[0009] In view of this, the present invention provides a sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation to solve the technical problems mentioned in the background technology.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] A sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation, including a row data multiplexing module, a dynamic non-zero value index module, a computing power dynamic allocation module and a multiplication and accumulation module;

[0012] The specific steps include:

[0013] S1. Input the feature map data of each input channel line by line into the buffer FIFO of the data multiplexing module, and gradually output the K-1 lines of feature map data after being cached by each FIFO;

[0014] S2. The dynamic non-zero value index module obtains 1 row of data of each input channel feature map and K-1 rows of feature map data gradually output after each FIFO cache and the weight data of each channel, and outputs the dynamic non-zero value index to the computing power dynamic allocation module after completing the splicing and index calculation;

[0015] S3. The computing power dynamic allocation module allocates the non-zero data pairs in the spliced ​​data stored in the data splicing module according to the dynamic non-zero value index output by the dynamic non-zero value index module, and outputs the allocation result to the multiplication and accumulation module;

[0016] S4. The multiplication and accumulation module performs multiplication and accumulation on the output of the dynamic allocation module of computing power, and adjusts the number of multiplication units according to the number of channels, convolution kernel size and sparsity to achieve the optimal utilization of the multiplication units.

[0017] Preferably, the number of buffer FIFOs is the order K of the convolution kernel minus 1, the feature map of each input channel is represented as an input feature map matrix, and the depth of each FIFO is set to the number of columns of the input feature map matrix.

[0018] Preferably, the dynamic non-zero value index module includes a data splicing module and an index calculation module;

[0019] The data splicing module combines the features of each input channel Figure 1 The row data and the K-1 row feature map data gradually output after each FIFO buffer are stored in register A, and the weight data of each channel is stored in register B. The data in register A is the local area data of the feature map corresponding to the convolution kernel; the data in register B is the convolution kernel data; the convolution kernel data of each channel and the local area data of the feature map corresponding to each channel are spliced ​​respectively;

[0020] The index calculation module performs mask indexing on the non-zero values ​​in the input feature map data and weight data of registers A and B in the data splicing module, and then performs position calculation to obtain a dynamic non-zero value index.

[0021] Preferably, the convolution kernel data of each channel and the local area data of the feature map corresponding to each channel are spliced ​​as follows:

[0022] Concatenate the flattened vectors of the C channels in sequence: the first row of data of the first convolution kernel is concatenated with the first row of data of the second convolution kernel, the first row of data of the second convolution kernel is concatenated with the first row of data of the third convolution kernel, and so on until they are concatenated with the first row of data of the Cth convolution kernel;

[0023] The second row of data of the first convolution kernel is concatenated with the second row of data of the second convolution kernel, and the second row of data of the second convolution kernel is concatenated with the second row of data of the third convolution kernel, until they are concatenated with the second row of data of the Kth convolution kernel;

[0024] Until the Kth row of data of the first convolution kernel is spliced ​​with the second row of data of the second convolution kernel, the Kth row of data of the second convolution kernel is spliced ​​with the Kth row of data of the third convolution kernel, until it is spliced ​​to the Kth row of data of the Cth convolution kernel, a two-dimensional matrix of size K×KC is obtained;

[0025] The weight data is stored in the register in the form of a one-dimensional array. The first row of the K×KC two-dimensional matrix is ​​stored in the array at positions 0-(KC-1), the second row is stored in the array at positions KC-2KC, and so on until the Kth row is stored in the array at positions (K-1)KC-(KKC-1);

[0026] The local area data of the feature maps corresponding to the corresponding channels are spliced ​​in the same way.

[0027] Preferably, the specific content of the index calculation module obtaining the dynamic non-zero value index is:

[0028] Use the Mask index to mark the inner product operation multiplication of the input feature map data and the weight data; if one of the input feature map data and the weight data is zero in the multiplication operation, the multiplication is considered an invalid operation, and the corresponding index is set to "0"; if both the input feature map data and the weight data are non-zero values, it is considered a valid operation, and the corresponding index is set to "1";

[0029] The dynamic non-zero value index is set to a vector, whose 0th bit is the number of non-zero data pairs, and the other bits are the positions of non-zero data pairs in sequence; sum all the values ​​in the Mask index to get the number of data pairs that need to be multiplied, and store it in the 0th bit of the array of the dynamic non-zero value index;

[0030] Read the values ​​of the Mask index one by one and record the position in the vector, determine whether the value is 1, if it is 1, store the position information in the dynamic non-zero value index in sequence, otherwise do not store the position information in the dynamic non-zero value index.

[0031] Preferably, the specific content of step S3 is:

[0032] S31. Divide the 0th bit data by the number of multiplication units using the dynamic non-zero value index to obtain the quotient q and the remainder r, which correspond to the number of cycles when the multiplication unit is fully loaded and the number of multiplication units that need to be calculated when not fully loaded;

[0033] S32. Record the calculation progress through the cycle count t. When an inner product operation calculation starts, count from 0 and record the number of cycles spent by the current inner product operation;

[0034] S33. Calculate the position base for obtaining the input feature map values ​​and weight data at different cycles, which is used to determine the addresses of the data in registers A and B at different cycles;

[0035] S34. Determine whether the current calculation state is within the effective range of the full load cycle or the non-full load cycle, and obtain the number of multiplication units performing the calculation;

[0036] S35. Obtain the feature map data and current weight data calculated by the current multiplication unit, and calculate the position information of each cycle of the inner product operation;

[0037] S36. Obtain the non-zero data at the corresponding multiplication positions in the input feature map data in register A and the weight data in register B according to the position information, and output them to the multiplication and accumulation module.

[0038] Preferably, in step S4, when the computing unit is fully loaded, D is accumulated MUThe product of a multiplication unit. When the computing unit is not fully loaded, the products of r multiplication units are accumulated. Starting from the first cycle of inner product calculation, the calculation results of each cycle are added together until the inner product calculation ends, obtaining a value of the output feature map.

[0039] Preferably, in step S32, when t is less than q, it indicates that the current inner product operation calculation is not completed. While each of the D MU multiplication units completes one calculation per cycle, t is incremented by 1; when t is equal to q, it indicates that one inner product operation calculation ends, t is cleared, and the number of cycles spent on the next inner product operation starts to be recorded;

[0040] In step S33, the position base Pos is: Pos = t × D MU ;

[0041] In step S34, when t < q, all multiplication units are in a fully loaded state, and at this time, the no-load problem does not need to be considered; when t == q && MU_num <= r, some multiplication units are in an idle state, and only r multiplication units are used to perform the calculation. MU_num is the multiplication unit number, and its value range is 1 - D MU .

[0042] Preferably, the specific content of step S35 is:

[0043] Let the currently obtained input feature map data be X, and let the currently obtained weight data be W. X = IFM[DNV[MU_num + Pos]], W = weight[DNV[MU_num + Pos]];

[0044] In the 0th cycle of the inner product operation, t = 0, Pos is 0, and the position information of DNV[1] - DNV[D MU will be taken out. In the 1st cycle of the inner product operation, t = 1, Pos is D MU , and the position information of DNV[1 + D MU - DNV[D MU + D MU will be taken out. In the 2nd cycle of the inner product operation, t = 2, Pos is 2 × D MU , and the position information of DNV[1 + 2 × D MU - DNV[D MU + 2 × D MU will be taken out, and so on;

[0045] In the last cycle of the inner product operation, when some multiplication units do not meet the conditions of t < q || (t == q && MU_num <= r), only the position information that meets the operation conditions is taken out, and the multiplication units that do not meet the conditions are made idle.

[0046] Preferably, when the convolution kernel and the corresponding local area of ​​the feature map perform inner product operation calculation, the multiplication unit utilization is:

[0047]

[0048] Among them, S local is the sparsity of the inner product matrix, K is the order of the convolution kernel, C is the number of channels, and D MU is the number of multiplication units, The round-up symbol.

[0049] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a sparse convolution acceleration method based on dynamic feature weight dual indexing and computing power allocation. Through dynamic non-zero value indexing, features and weights can be indexed at the same time, and the calculation time and memory requirements of the index are reduced. A computing power dynamic allocation strategy is adopted to perform computing power allocation operations on data after zero jumps, and the data after zero jumps of multiple channel convolutions are concentratedly allocated to a group of multiplication units, which reduces the difficulty of pairing non-zero data. The number of multiplication units can be adjusted as needed according to the number of channels, convolution kernel size and sparsity to achieve optimal utilization of multiplication units and avoid idle computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0051] Figure 1 A schematic diagram of a sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation provided by the present invention;

[0052] Figure 2 A schematic diagram of the data flow of the buffer FIFO provided by the present invention;

[0053] Figure 3 A schematic diagram of the process of updating stored data by the data reuse module provided by the present invention;

[0054] Figure 4 A schematic diagram of the splicing operation of the convolution kernel data and the local area data of the feature map provided by the present invention;

[0055] Figure 5 A schematic diagram of index calculation provided by the present invention;

[0056] Figure 6 A schematic diagram of the circuit implementation logic of the computing power dynamic allocation module provided by the present invention;

[0057] Figure 7 A schematic diagram of allocating data through indexes provided by the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The embodiment of the present invention discloses a sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation, such as Figure 1 , including a row data multiplexing module, a dynamic non-zero value indexing module, a computing power dynamic allocation module and a multiplication and accumulation module;

[0060] The specific steps include:

[0061] S1. Input the feature map data of each input channel line by line into the buffer FIFO of the data multiplexing module, and gradually output the K-1 lines of feature map data after being cached by each FIFO;

[0062] S2. The dynamic non-zero value index module obtains 1 row of data of each input channel feature map and K-1 rows of feature map data gradually output after each FIFO cache and the weight data of each channel, and outputs the dynamic non-zero value index to the computing power dynamic allocation module after completing the splicing and index calculation;

[0063] S3. The computing power dynamic allocation module allocates the non-zero data pairs in the spliced ​​data stored in the data splicing module according to the dynamic non-zero value index output by the dynamic non-zero value index module, and outputs the allocation result to the multiplication and accumulation module;

[0064] S4. The multiplication and accumulation module performs multiplication and accumulation on the output of the dynamic allocation module of computing power, and adjusts the number of multiplication units according to the number of channels, convolution kernel size and sparsity to achieve the optimal utilization of the multiplication units.

[0065] In order to further implement the above technical solution, the number of buffer FIFOs is the order K of the convolution kernel minus 1, the feature map of each input channel is represented as an input feature map matrix, and the depth of each FIFO is set to the number of columns of the input feature map matrix.

[0066] In this embodiment, if Figure 2The buffer FIFO is first-in-first-out, and the working method is as follows: starting from the 1st channel to the Cth channel, the input feature map matrix of each channel is input into the FIFO row by row; starting from the first row, each row of data of the input feature map matrix is ​​input into the 1st FIFO in sequence, and is also output to the dynamic non-zero value index module; when the 1st FIFO is full, its output data is input into the 2nd FIFO in sequence, and is also output to the dynamic non-zero value index module; when the 2nd FIFO is full, the data therein is input into the 3rd FIFO in sequence, and is also output to the dynamic non-zero value index module, and so on.

[0067] In order to further implement the above technical solution, the dynamic non-zero value index module includes a data splicing module and an index calculation module;

[0068] Data splicing module, such as Figure 3 , each input channel feature Figure 1 The row data and the K-1 row feature map data gradually output after each FIFO buffer are stored in register A, and the weight data of each channel is stored in register B. The data in register A is the local area data of the feature map corresponding to the convolution kernel; the data in register B is the convolution kernel data; the convolution kernel data of each channel and the local area data of the feature map corresponding to each channel are spliced ​​respectively;

[0069] In this embodiment, one row of data of the characteristic graph of C input channels input externally is stored in each channel in each cycle. After K data are stored, the data input in the current cycle replaces the data of the previous K cycle. The C input channels store a total of C×K data, which are stored in register A. Each FIFO output stores only one data in each cycle. After K data are stored, the data input in the current cycle replaces the data of the previous K cycle. The C×(K-1) FIFO outputs store a total of C×(K-1)×K data, which are stored in register A.

[0070] The weight data of the C channels of external input, each channel only stores 1 data in each cycle, and the C input channels store a total of C×K×K data, which are stored in register B;

[0071] The index calculation module performs mask indexing on the non-zero values ​​in the input feature map data and weight data of registers A and B in the data splicing module, and then performs position calculation to obtain the dynamic non-zero value DNV index.

[0072] In order to further implement the above technical solution, Figure 4 , the convolution kernel data of each channel and the local area data of the feature map corresponding to each channel are spliced ​​as follows:

[0073] Concatenate the flattened vectors of the C channels in sequence: the first row of data of the first convolution kernel is concatenated with the first row of data of the second convolution kernel, the first row of data of the second convolution kernel is concatenated with the first row of data of the third convolution kernel, and so on until they are concatenated with the first row of data of the Cth convolution kernel;

[0074] The second row of data of the first convolution kernel is concatenated with the second row of data of the second convolution kernel, and the second row of data of the second convolution kernel is concatenated with the second row of data of the third convolution kernel, until they are concatenated with the second row of data of the Kth convolution kernel;

[0075] Until the Kth row of data of the first convolution kernel is spliced ​​with the second row of data of the second convolution kernel, the Kth row of data of the second convolution kernel is spliced ​​with the Kth row of data of the third convolution kernel, until it is spliced ​​to the Kth row of data of the Cth convolution kernel, a two-dimensional matrix of size K×KC is obtained;

[0076] The weight data is stored in the register in the form of a one-dimensional array. The first row of the K×KC two-dimensional matrix is ​​stored in the array at positions 0-(KC-1), the second row is stored in the array at positions KC-2KC, and so on until the Kth row is stored in the array at positions (K-1)KC-(KKC-1);

[0077] The local area data of the feature maps corresponding to the corresponding channels are spliced ​​in the same way.

[0078] In order to further implement the above technical solution, Figure 5 , the specific content of the index calculation module to obtain the dynamic non-zero value index is:

[0079] Use the Mask index to mark the inner product operation multiplication of the input feature map data and the weight data; if one of the input feature map data and the weight data is zero in the multiplication operation, the multiplication is considered an invalid operation, and the corresponding index is set to "0"; if both the input feature map data and the weight data are non-zero values, it is considered a valid operation, and the corresponding index is set to "1";

[0080] In this embodiment, a total of n pairs of data are judged, the input feature map data and weight data to be multiplied are a pair, and n is the number of elements in the concatenated K×KC matrix;

[0081] The dynamic non-zero value index is set to a vector, whose 0th bit is the number of non-zero data pairs, and the other bits are the positions of non-zero data pairs in turn; sum all the values ​​in the Mask index to get the data pairs N that need to be multiplied nz The number of and stored in the 0th position of the array of dynamic non-zero value index;

[0082] Read the values ​​of the Mask index one by one and record the position in the vector, determine whether the value is 1, if it is 1, store the position information in the dynamic non-zero value index in sequence, otherwise do not store the position information in the dynamic non-zero value index.

[0083] In order to further implement the above technical solution, Figure 6 , the specific content of step S3 is:

[0084] S31. Use the dynamic non-zero value to index the 0th bit data divided by the number of multiplication units MU D MU The quotient q and remainder r are obtained, which correspond to the number of cycles when the multiplication unit is fully loaded and the number of multiplication units that need to be calculated when not fully loaded;

[0085] S32. Record the calculation progress through the cycle count t. When an inner product operation calculation starts, count from 0 and record the number of cycles spent by the current inner product operation;

[0086] S33. Calculate the position base Pos for obtaining the input feature map values ​​and weight data at different cycles, which is used to determine the addresses of the data in registers A and B at different cycles;

[0087] S34. Determine whether the current calculation state is within the effective range of the full load cycle or the non-full load cycle, and obtain the multiplication unit that performs the calculation;

[0088] S35. Obtain the feature map data and current weight data calculated by the current multiplication unit, and extract the position information of each cycle of the inner product operation;

[0089] S36. Obtain the non-zero data at the corresponding multiplication positions in the input feature map data in register A and the weight data in register B according to the position information, and output them to the multiplication and accumulation module.

[0090] In order to further implement the above technical solution, in step S4, when the computing unit is fully loaded, D is accumulated. MU When the computing unit is not fully loaded, the products of r multiplication units are accumulated, starting from the first cycle of the inner product calculation. The calculation results of each cycle are added until the inner product calculation is completed to obtain a value of the output feature map.

[0091] In order to further implement the above technical solution, in step S32, when t is less than q, it means that the current inner product operation calculation is not completed, D MU Each multiplication unit completes a calculation in each cycle and t is incremented by 1; when t is equal to q, it means that the calculation of an inner product operation is completed, t is reset to zero, and the number of cycles spent on the next inner product operation begins to be recorded;

[0092] In step S33, the position cardinality Pos is: Pos = t × D MU ;

[0093] In step S34, when t < q, all multiplication units are in the full-load state, and there is no need to consider the no-load problem at this time; when t == q && MU_num <= r, some multiplication units are in the idle state, and only r multiplication units are used to perform the calculation. MU_num is the multiplication unit number, and its value range is 1 - D MU 。

[0094] To further implement the above technical solution, the specific content of step S35 is as follows:

[0095] Let the currently acquired input feature map data be X, and let the currently acquired weight data be W. X = IFM[DNV[MU_num + Pos]], W = weight[DNV[MU_num + Pos]];

[0096] In the 0th cycle of the inner product operation, t = 0, Pos is 0, and the position information of DNV[1] - DNV[D MU will be taken out. In the 1st cycle of the inner product operation, t = 1, Pos is D MU , and the position information of DNV[1 + D MU - DNV[D MU + D MU will be taken out. In the 2nd cycle of the inner product operation, t = 2, Pos is 2 × D MU , and the position information of DNV[1 + 2 × D MU - DNV[D MU + 2 × D MU will be taken out, and so on;

[0097] In the last cycle of the inner product operation, when some multiplication units do not meet the conditions of t < q || (t == q && MU_num <= r), only the position information that meets the operation conditions is taken out, and the multiplication units that do not meet the conditions are made idle.

[0098] To further implement the above technical solution, the number N of non-zero value data pairs in the process of the convolution kernel performing the inner product operation with the corresponding local region of the feature map nz is:

[0099] N nz = S local K 2 C

[0100] Among them, S local represents the proportion of non-zero values in C local regions of the feature map when the convolution kernel slides to a specific position in the convolution operation. K is the convolution kernel order, and C is the channel number size;

[0101] The number of cycles T of the convolution kernel performing the inner product operation with the corresponding local region of the feature map total is:

[0102]

[0103] Among them, D MU represents the number of multiplication units, is the round-up symbol, when When it is not an integer, the calculation cycle needs to be rounded up, which means that in the last cycle, even if the multiplication unit is not fully loaded, a cycle of calculation must be performed, and the multiplication unit will be partially idle at this time;

[0104] The multiplication unit utilization U of the inner product operation calculation process performed by the convolution kernel and the corresponding local area of ​​the feature map is:

[0105]

[0106] Among them, T total D MU Indicates that all multiplication units are in T total The total number of working times in a cycle, including effective and invalid calculations, N nz It represents the number of non-zero value data pairs that need to be calculated, and also represents the actual effective calculation times of the multiplication unit;

[0107] Then, when the convolution kernel and the corresponding local area of ​​the feature map perform inner product operations, the multiplication unit utilization is:

[0108]

[0109] Among them, S local is the sparsity of the inner product matrix, K is the order of the convolution kernel, C is the number of channels, and D MU is the number of multiplication units, is the rounding symbol;

[0110] Since the inner product matrix is ​​sparse S local , the convolution kernel size K and the number of channels C are fixed, so the number of multiplication units D can be changed MU Maximize the multiplication unit utilization U.

[0111] In another embodiment, the size of the input feature map is set to 28×28, the size of the convolution kernel is set to 3×3, the input channel is set to 4, and the number of multiplication units is set to 8;

[0112] The data multiplexing module uses multiple first-in-first-out buffers FIFO to receive input feature map data from 4 channels. The FIFO module can be generated using IP core RAM-based Shift Register to set parameters, or the module can be written according to logic;

[0113] The number of FIFOs is the order K of the convolution kernel minus 1. If the convolution kernel size is 3×3, then 2 FIFOs are required.

[0114] The input feature map of each channel is represented as an input feature map matrix;

[0115] The depth of each FIFO is set to the number of columns of the input feature map matrix. The size of the input feature map is 28×28, so the FIFO depth is set to 28, and each FIFO stores 28 data;

[0116] Starting from the first channel to the fourth channel, the input feature map matrix of each channel is input into the FIFO row by row; starting from the first row, each row of the input feature map matrix is ​​sequentially input into the first FIFO, and is also output to the dynamic non-zero value index module; when the first FIFO is full, its output data is sequentially input into the second FIFO, and is also output to the dynamic non-zero value index module; when the second FIFO is full, the data therein is sequentially input into the third FIFO, and is also output to the dynamic non-zero value index module;

[0117] The dynamic non-zero value index module includes a data splicing module and an index calculation module, which receives 1 line of data gradually input from the external 4 input channel feature maps and 2 lines of feature map data gradually output from the data multiplexing module; at the same time, it receives the weight data of the 4 channels input from the external; after completing the index calculation, it outputs the DNV index to the dynamic allocation module;

[0118] The data splicing module receives the 4 input channel features of external input Figure 1 1 row of feature map data output by the row data and data multiplexing module; 1 row of feature map data of the 4 input channels input externally, each channel stores only 1 data per cycle, after 3 data are stored, the data input in the current cycle will replace the data of the previous 3 cycles, 4 input channels store a total of 12 data, stored in register A, each FIFO output stores only 1 data per cycle, after 3 data are stored, the data input in the current cycle will replace the data of the previous 3 cycles, 8 FIFO outputs store a total of 24 data, stored in register A, register A stores a total of 36 data, the data in register A is the local area data of the feature map corresponding to the convolution kernel;

[0119] Receive the input channel feature map data and the weight data of the 4 channels input from the outside at the same time. Each channel only stores 1 data in each cycle. The 4 input channels store a total of 36 data in register B. The data in register B is the convolution kernel data.

[0120] Register A and register B can be implemented using a lookup table;

[0121] The convolution kernel data of the four channels and the local area data of the feature maps corresponding to the four channels are spliced ​​respectively; the flattened vectors of the four channels are spliced ​​in sequence, the first row of data of the first convolution kernel is spliced ​​with the first row of data of the second convolution kernel, the first row of data of the second convolution kernel is spliced ​​with the first row of data of the third convolution kernel, the first row of data of the third convolution kernel is spliced ​​with the first row of data of the fourth convolution kernel; the second row of data of the first convolution kernel is spliced ​​with the second convolution kernel The second row of data, the second row of data of the second convolution kernel is spliced ​​with the second row of data of the third convolution kernel, the second row of data of the third convolution kernel is spliced ​​with the second row of data of the fourth convolution kernel; the third row of data of the first convolution kernel is spliced ​​with the third row of data of the second convolution kernel, the third row of data of the second convolution kernel is spliced ​​with the third row of data of the third convolution kernel, and the third row of data of the third convolution kernel is spliced ​​with the third row of data of the fourth convolution kernel; a two-dimensional matrix of size 3×12 is obtained;

[0122] The weight data is stored in the register in the form of a one-dimensional array. The first row of the 3×12 two-dimensional matrix is ​​stored in the array at positions 0-11, the second row is stored in the array at positions 12-23, and so on until the third row is stored in the array at positions 24-35.

[0123] Similarly, the corresponding local regions of the feature maps are spliced;

[0124] The index calculation module performs mask indexing on the non-zero values ​​in the input feature map data and weight data of registers A and B in the data splicing module, and then inputs the mask index into the position calculation module to obtain a dynamic non-zero value index.

[0125] The index calculation module first uses the Mask index to mark the inner product operation multiplication; a total of 36 pairs of data are judged, and the input feature map data and weight data that need to be multiplied are a pair; if one of the input feature map data and weight data in the multiplication operation is zero, the multiplication is considered an invalid operation, and the corresponding index is set to "0"; if both the input feature map data and the weight data are non-zero values, it is considered a valid operation, and the corresponding index is set to "1";

[0126] The dynamic non-zero value index is set to a vector whose 0th bit is the number of non-zero data pairs, and the other bits are the positions of non-zero data pairs in sequence;

[0127] Sum all the values ​​in the Mask index to get the data pair N that needs to be multiplied. nz The number of and store it in the 0th position of the array with dynamic non-zero index;

[0128] The position calculation module reads the value of the Mask index one by one, records the position in the vector, and then determines whether its value is 1. If it is 1, the position information is stored in the dynamic non-zero value index in sequence. If it is not 1, the position information is not stored in the dynamic non-zero value index.

[0129] The computing power dynamic allocation module allocates non-zero data pairs among the concatenated data stored in register A and register B in the data concatenation module according to the dynamic non-zero value index output by the dynamic non-zero value indexing module, and outputs the allocation result to the multiply-accumulate module;

[0130] The quotient q and the remainder r obtained by dividing the 0th bit data of the dynamic non-zero value by the number of multiplication units 8 respectively correspond to the number of cycles when the multiplication units are fully loaded and the number of multiplication units that need to perform calculations when they are not fully loaded;

[0131] In this embodiment, the divider can use a lookup table to accelerate division, that is, all quotients and remainders are pre-calculated and stored in the lookup table, and the lookup table returns the quotient and the remainder through direct indexing. This method can save resources better than a general divider and avoid timing problems.

[0132] Specifically: the value of the dynamic non-zero value is 14, calculate Get is the floor function symbol, calculate that there is 1 full-load cycle, 14 % 8 = 6 to get r, and calculate that 6 multiplication units are used in the non-full-load cycle.

[0133] The cycle count t records the calculation progress; at the start of a dot product operation calculation, it starts counting from 0 and records the number of cycles that have elapsed for the current dot product operation; when t is less than q, it means that the current dot product operation calculation is not completed, and while the 8 multiplication units complete one calculation per cycle, t is incremented by 1; when t is equal to q, it means that a dot product operation calculation is completed, t is cleared, and the number of cycles that have elapsed for the next dot product operation starts to be recorded;

[0134] Then, a position base Pos is calculated for the input feature map values and weight data at different cycles, Pos = t × 8, which is used to determine the addresses of the data in register A and register B at different cycles;

[0135] Judge whether it is currently within the valid range of the full-load cycle or the non-full-load cycle. When t < q, all multiplication units are in the full-load state, and at this time, the no-load problem does not need to be considered; the 0th cycle is in the full-load state, and the non-zero data pairs are allocated according to the number of bits of the array. The 1st to 8th bits of the array are allocated to the 1st to 8th multiplication units as the calculation for the 1st cycle, and 8 groups of non-zero data pairs are input; when (t == q && MU_num <= r), some multiplication units are in the idle state, and only r multiplication units are used to perform the calculation; the 1st cycle is in the non-full-load state. For the 9th to 14th bits, the 1 to 6 calculated by taking the remainder are allocated to the 1st to 6th multiplication units in the 2nd cycle, and 6 groups of non-zero data pairs are input. MU_num is the multiplication unit number, and its value range is 1 - 8. The result is as Figure 7 shown;

[0136] After determining whether the current computing state is in the full-load cycle or the non-full-load cycle, obtain the data calculated by the current multiplication unit. Let the currently acquired input feature map data be X, and the currently acquired weight data be W.

[0137] X = IFM[DNV[MU_num + 8]], W = weight[DNV[MU_num + 8]]; At the 0th cycle of the inner product operation, t = 0, Pos is 0, and the position information of DNV[1] - DNV[8] will be fetched. At the 1st cycle of the inner product operation, t = 1, Pos is 8. There is a situation where the 7th and 8th multiplication units do not meet the condition of t < q || (t == q && MU_num <= r). The position information of DNV[9] - DNV

[14] will be fetched, and the position information of DNV

[15] - DNV

[16] will not be fetched, and the 7th and 8th multiplication units will be set to idle.

[0138] Obtain the non-zero data at the corresponding multiplication positions of the input feature map data in register A and the weight data in register B according to the position information, and output it to the multiply-accumulate module.

[0139] The multiply-accumulate module performs multiply-accumulation on the output of the computing power dynamic allocation module. When the computing unit is in the full-load state, the products of 8 multiplication units are accumulated. When the computing unit is in the non-full-load state, the products of r multiplication units are accumulated. Starting from the first cycle of the inner product calculation, the calculation results of each cycle are added together until the inner product calculation ends, obtaining a value of the output feature map; when performing the inner product operation between the convolution kernel and the corresponding local area of the feature map, the inner product results of multiple input channels in the same output channel can be directly added; therefore, during the calculation process, the results can be accumulated step by step; in this example, there are 14 multiplication results for 4 input channels. At the 0th cycle, the sum of eight numbers is calculated. At the 1st cycle, it is accumulated on the result of the 0th cycle, and then 6 more numbers are added; obtaining the value of one inner product, that is, a value of the output feature map.

[0140] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0141] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation, characterized in that: It includes row data multiplexing module, dynamic non-zero value index module, computing power dynamic allocation module and multiplication and accumulation module; The specific steps include: S1. Input the feature map data of each input channel line by line into the buffer FIFO of the data multiplexing module, and gradually output the K-1 lines of feature map data after being cached by each FIFO; S2. The dynamic non-zero value index module obtains 1 row of data of each input channel feature map and K-1 rows of feature map data gradually output after each FIFO cache, as well as the weight data of each channel, where K is the convolution kernel order, and outputs the dynamic non-zero value index DNV to the computing power dynamic allocation module after completing the splicing and index calculation; S3. The computing power dynamic allocation module allocates the non-zero data pairs in the spliced ​​data stored in the data splicing module according to the dynamic non-zero value index output by the dynamic non-zero value index module, and outputs the allocation result to the multiplication and accumulation module; S4. The multiplication and accumulation module performs multiplication and accumulation on the output of the dynamic allocation module of computing power, and adjusts the number of multiplication units according to the number of channels, convolution kernel size and sparsity to achieve the optimal utilization of the multiplication units.

2. According to claim 1, a sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation is characterized in that: The number of buffer FIFOs is the order K of the convolution kernel minus 1. The feature map of each input channel is represented as an input feature map matrix, and the depth of each FIFO is set to the number of columns of the input feature map matrix.

3. According to claim 1, a sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation is characterized in that: The dynamic non-zero value index module includes a data splicing module and an index calculation module; The data splicing module stores 1 row of feature map data of each input channel and K-1 rows of feature map data gradually output after being buffered by each FIFO into register A, and stores the weight data of each channel into register B. The data in register A is the local area data of the feature map corresponding to the convolution kernel; the data in register B is the convolution kernel data; the convolution kernel data of each channel and the local area data of the feature map corresponding to each channel are spliced ​​respectively; The index calculation module performs mask indexing on the non-zero values ​​in the input feature map data and weight data of registers A and B in the data splicing module, and then performs position calculation to obtain a dynamic non-zero value index.

4. According to claim 3, a sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation is characterized in that: The contents of splicing the convolution kernel data of each channel and the local area data of the feature map corresponding to each channel are as follows: Concatenate the flattened vectors of the C channels in sequence: the first row of data of the first convolution kernel is concatenated with the first row of data of the second convolution kernel, the first row of data of the second convolution kernel is concatenated with the first row of data of the third convolution kernel, and so on until they are concatenated with the first row of data of the Cth convolution kernel; The second row of data of the first convolution kernel is concatenated with the second row of data of the second convolution kernel, and the second row of data of the second convolution kernel is concatenated with the second row of data of the third convolution kernel, until they are concatenated with the second row of data of the Kth convolution kernel; Until the Kth row of data of the first convolution kernel is spliced ​​with the second row of data of the second convolution kernel, the Kth row of data of the second convolution kernel is spliced ​​with the Kth row of data of the third convolution kernel, until it is spliced ​​to the Kth row of data of the Cth convolution kernel, a two-dimensional matrix of size K×KC is obtained; The weight data is stored in the register in the form of a one-dimensional array. The first row of the K×KC two-dimensional matrix is stored in the 0-(KC-1) bits of the array, the second row is stored in the KC-2KC bits of the array, and so on until the Kth row is stored in the (K-1)KC-(KKC-1) bits of the array; Similarly, the local region data of the feature maps corresponding to each channel is concatenated.

5. According to claim 3, a sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation is characterized in that: The specific content obtained by the index calculation module for the dynamic non-zero value index is as follows: Use the Mask index to mark the multiplication operation of the inner product of the input feature map data and the weight data; if one of the input feature map data and the weight data in the multiplication operation is zero, the multiplication is regarded as an invalid operation, and the corresponding index is set to "0"; if both the input feature map data and the weight data are non-zero values, it is regarded as a valid operation, and the corresponding index is set to "1"; The dynamic non-zero value index is set to a vector. Its 0th bit is the number of non-zero data pairs, and the other bits are the positions of the non-zero data pairs in sequence; sum all the values in the Mask index to obtain the number of data pairs that need to perform multiplication operations, and store it in the 0th bit of the array of the dynamic non-zero value index; Read the values of the Mask index one by one and record the positions in the vector. Determine whether the value is 1. If it is 1, store the position information in the dynamic non-zero value index in sequence, otherwise do not store the position information in the dynamic non-zero value index.

6. According to claim 5, a sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation is characterized in that: The specific content of step S3 is as follows: S31. Divide the data in the 0th bit of the dynamic non-zero value index by the number of multiplication units to obtain the quotient q and the remainder r, which correspond to the number of cycles when the multiplication units are full and the number of multiplication units that need to be calculated when they are not full, respectively; S32. Record the calculation progress through the cycle count t. When the calculation of an inner product operation starts, start counting from 0 and record the number of cycles that the current inner product operation has spent; S33. Calculate the position base for the input feature map values and the weight data at different cycles, which is used to determine the addresses of the data in register A and register B at different cycles; S34. Determine whether the current calculation state is within the valid range of the full-load cycle or the non-full-load cycle to obtain the number of multiplication units that perform the calculation; S35. Obtain the feature map data and the weight data calculated by the current multiplication unit, and calculate the position information at each cycle of the inner product operation; S36. According to the position information, obtain the data that is non-zero at the corresponding multiplication positions of the input feature map data in register A and the weight data in register B, and output it to the multiply-accumulate module.

7. The sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation according to claim 6 is characterized in that: In step S4, when the computing unit is fully loaded, D MU When the computing unit is not fully loaded, the products of r multiplication units are accumulated, starting from the first cycle of the inner product calculation. The calculation results of each cycle are added until the inner product calculation is completed to obtain a value of the output feature map.

8. The sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation according to claim 6, characterized in that: In step S32, when t is less than q, it means that the current inner product operation calculation is not completed, D MU Each multiplication unit completes a calculation in each cycle and t is incremented by 1; when t is equal to q, it means that the calculation of an inner product operation is completed, t is reset to zero, and the number of cycles spent on the next inner product operation begins to be recorded; In step S33, the position cardinality Pos is: Pos = t × D MU ; In step S34, when t < q, all multiplication units are in the fully loaded state, and there is no need to consider the idle problem at this time; when t == q && MU_num <= r, some multiplication units are in the idle state, and only r multiplication units are used to perform calculations. MU_num is the multiplication unit number, and its value range is 1 - D MU .

9. The sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation according to claim 6, characterized in that: The specific content of step S35 is as follows: Let the currently obtained input feature map data be X, and the currently obtained weight data be W. X = IFM[DNV[MU_num+Pos]], W = weight[DNV[MU_num+Pos]]; In the 0th cycle of the inner product operation, t = 0, Pos is 0, and DNV[1]-DNV[D MU ], in the first cycle of the inner product operation, t = 1, Pos is D MU , DNV[1+D MU ]-DNV[D MU +D MU ], in the second cycle of the inner product operation, t = 2, Pos is 2 × D MU , DNV[1+2×D MU ]-DNV[D MU +2×D MU ] location information, and so on; At the last cycle of the inner product operation, when some multiplication units do not meet the condition of t<q||(t==q&&MU_num<=r), only take out the position information that meets the operation condition, and idle the multiplication units that do not meet the condition.

10. The sparse convolution acceleration method based on dynamic feature weight dual index and computing power allocation according to claim 1, characterized in that: When the convolution kernel performs an inner product operation with the corresponding local region of the feature map, the utilization rate of the multiplication units is: Among them, S local is the sparsity of the inner product matrix, K is the order of the convolution kernel, C is the number of channels, and D MU is the number of multiplication units, The symbol for rounding up.

Images