A method and apparatus for convolution computation

Abstract

The patent discloses a convolution calculation method and device for optimizing energy consumption and efficiency in existing convolution calculation process. The main technical scheme of the patent is that the feature data and weight data are input to the calculation unit through the internal memory according to the block, and the input end queue of the multiplier is dynamically allocated. The patent optimizes the calculation process in the calculation unit, utilizes the channel data characteristics, makes the output results in a single multiplier group coherent, the input end weight data of each multiplier in the same multiplier group belongs to different convolution kernels, so that data competition does not occur, the adder can be directly operated, and the multiplier output does not need to wait in the queue. In addition, the convolution kernel size, form and data reading mode have no influence on the calculation process of the patent, so the corresponding calculation of different strides, different padding forms, different forms of convolution (such as various deformation convolutions) has good adaptability.

Description

Technical Field

[0001] This invention relates to the field of computer systems based on specific computational models, and more particularly to a convolution computation method and apparatus. Background Technology

[0002] In deep learning computation, the convolution operation between weights and input features requires a large number of multiply-accumulate (MAC) operations. In traditional computation methods, when the computation process is not optimized, the following problems will occur: 1. Input feature data, weight data, and output data will be repeatedly read and written, resulting in waste of storage space, bandwidth, and a large amount of energy wasted due to data movement; 2. Low computational parallelism leads to slow computation speed and low computing power; 3. There is competition between data, which leads to computational blockage and reduces computing power.

[0003] For optimizing multi-channel convolution in deep learning computation, the following existing solutions exist:

[0004] Existing calculation method 1: Follow the calculation order and logic of convolution operation completely. Each time, calculate the data in the same convolution kernel matrix. Taking a 3×3 convolution kernel size as an example, when the data is not compressed, first calculate the multiplication of the 9 related data in the current convolution kernel and add the multiplication results together.

[0005] However, traditional computing methods cannot directly use data compression to accelerate the process. This is because when the data is compressed, each computation only calculates the same convolution kernel matrix, and it is impossible to completely skip non-zero elements. At most, zero-valued data can be ignored, thus saving computational power consumption.

[0006] With attachment Figure 1 Taking the data shown as an example, after data compression, the calculation only requires 3 multiplications and 2 additions:

[0007] output = f1×w1 + f5×w5 + f6×w6

[0008] However, due to the uncertainty of data compression, for data with different sparsity levels, there might be 9 multiplications and 8 additions, 8 multiplications and 7 additions, or even 2 multiplications and 1 addition. To ensure computational versatility, the hardware for multipliers and adders still requires 9 and 8 units respectively. Therefore, using additional... Figure 1 In the compressed data, the multiplier utilization rate is only 3 / 9 and the adder utilization rate is only 2 / 8. Although data compression reduces power consumption, there is a huge waste of hardware and computing power.

[0009] Existing calculation method two: fully utilizes the multiplier, performing 9 multiplication operations in a single calculation. However, due to data compression, the correlation of the 9 multiplication results is uncertain. Therefore, addition operations cannot be directly performed after multiplication operations, and the multiplier output may need to enter a queue to wait. For example, see [link to example] for input feature data compression. Figure 2 :

[0010] In a single calculation, 9 multiplications are performed, and 9 multiplication results are output:

[0011] The convolution kernels are located at the initial positions: f1×w1, f2×w2, f3×w9

[0012] The convolution kernel moves to the right for the first time: f2×w1, f3×w8, f4×w6

[0013] The convolution kernel moves to the right for the second time: f3×w7, f4×w5, f2×w3

[0014] Among them, the three results f1×w1, f2×w2, and f3×w9 are correlated; the three results f2×w1, f3×w8, and f4×w6 are correlated; and the three results f3×w7, f4×w5, and f2×w3 are correlated, as shown in the following formula:

[0015] out1 = f1 × w1 + f2 × w2 + f3 × w9

[0016] out2 = f2 × w1 + f3 × w8 + f4 × w6

[0017] out3 = f3 × w7 + f4 × w5 + f2 × w3

[0018] Taking f1×w1, f2×w2, and f3×w9 as an example, the three multiplication outputs are added together. Since the correlation between each multiplication result is uncertain, the relevant results can only be placed in a memory queue and accumulated multiple times to obtain the final result, rather than being parallelized. The calculation of out1 is actually done through two addition operations, that is, two addition cycles. Each addition operation consists of only one addition operation, as shown in the following formula:

[0019] Period 1: out cache =f1×w1+f2×w2

[0020] Period 2: out1 = out cache +f3×w9

[0021] Therefore, the time complexity of multiplication and addition is mismatched. For every calculation performed by the multiplier, the adder may need to perform 1 to 9 calculations. This can lead to a large backlog of data at the adder's input, wasting computation time. The only solution is to temporarily store the data and wait for each calculation in turn. When memory space is insufficient, the multiplication result cannot be stored, the multiplier stops calculating, and thus becomes blocked. Furthermore, to ensure computational versatility (the data compression is uncertain, potentially resulting in operations such as 9 multiplications and 8 additions, 9 multiplications and 7 additions, ... 9 multiplications and 1 addition), the adder hardware still requires 8 units, i.e. Figure 2 After data compression, the adder utilization rate is 3 / 9. Although the power consumption is reduced, there is a huge waste of hardware and computing power.

[0022] The third existing calculation method is to make full use of multipliers and optimize the calculation process so that the outputs of all multipliers are independent of each other, thus avoiding data races. Adders can operate directly, and multiplier outputs do not need to wait in a queue. However, this method requires a large amount of output addressing hardware and extremely complex output addressing logic, which requires huge hardware resources and is too costly. Summary of the Invention

[0023] This invention provides a convolution calculation method and apparatus to address the problems of low hardware efficiency and high resource consumption in existing deep learning calculation processes.

[0024] This invention provides a convolution calculation method for image processing, characterized in that feature data and weight data are input to the calculation unit in blocks via internal memory, and the input queue of the multiplier is dynamically allocated; the feature data is image feature data, and the weight data is obtained after training a neural network model.

[0025] Preferably, the internal memory stores the blocks sequentially into the queue at the input of the multiplier according to the channel order.

[0026] Preferably, the dynamic allocation of the multiplier input queue specifically includes: when the multiplier input queue meets specific conditions, data is sequentially moved from other multiplier input queues for multiplication calculation.

[0027] Preferably, the specific condition is that the multiplier input queue is empty.

[0028] Preferably, data is moved from the input queue of the adjacent multiplier first.

[0029] Preferably, the block input makes the output results of a single multiplier group coherent, and the input weight data of each multiplier in the same multiplier group belong to different channels of the same convolution kernel.

[0030] Preferably, both the weight data and the feature data are sparse data.

[0031] Preferably, the feature data and weight data are preprocessed and filtered to select valid data before being input into the internal memory.

[0032] Preferably, the calculation unit skips invalid calculations based on the content of the input data.

[0033] Preferably, the method is used to process speech and text, the feature data is speech data and text data, and the weight data is obtained after the speech and text have been trained by a neural network model.

[0034] This invention provides a convolution calculation device, comprising: an external memory, an internal memory, and a calculation unit;

[0035] The feature data and weight data in the external memory are input into the internal memory in blocks;

[0036] The computing unit includes a computing module, a storage module, and a control module;

[0037] The computation module includes multiple multiplier groups and an adder tree, wherein the multiplier groups include multiple multipliers;

[0038] The storage module is connected to the internal memory and the computing module, and is used to cache the input data of the computing module;

[0039] The control module connects the computing module and the storage module, and is used to control the input of the multiplier and the dynamic distribution of the input queue.

[0040] The present invention has the following beneficial effects:

[0041] 1. Both input weight data and input feature data support compression, making them suitable for both sparse and dense data operations;

[0042] 2. Employ a multi-level storage system that combines external memory, internal memory, and registers to reduce power consumption caused by repeated data reads and writes;

[0043] 3. Optimize the data flow calculation order and reduce the hardware size of the internal memory;

[0044] 4. By optimizing the computation process in the computing unit and utilizing the characteristics of channel data, the output results within a single multiplier group are made coherent, and the input weight data of each multiplier in the same multiplier group belong to different convolution kernels. Therefore, there will be no data competition, the adder can operate directly, and the multiplier output does not need to queue.

[0045] 5. Since the outputs of a single multiplier group are coherent, adding is performed using an adder tree, which significantly reduces adder power consumption and data transfer power consumption.

[0046] 6. The calculation order of the output results is consistent with the final storage data location order, without the need for additional addressing hardware and complex logic calculations, making it easy to implement output result addressing;

[0047] 7. Because this patent splits the data into single-point operations, the multiplier group only calculates a portion of the output value each time. The size, form, and data reading method of the convolution kernel have no impact on this calculation process. Therefore, it has good adaptability to calculations with different strides, different padding forms, and different forms of convolution (such as various deformed convolutions).

[0048] 8. By redistributing the valid data among register groups, the valid data and computation are distributed as evenly as possible to each multiplier, thereby improving the utilization of the multipliers and avoiding the phenomenon that one or a few multipliers need to continuously perform calculations while most other multipliers are idle and waiting, wasting hardware resources. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the calculation process of an existing convolution calculation method in the background art of this invention;

[0051] Figure 2 This is a schematic diagram of the calculation process of the existing convolution calculation method 2 in the background art of this invention;

[0052] Figure 3 This is a schematic diagram of the calculation process combining a convolution calculation device and a method in one embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the computing unit structure and the dynamic allocation method in one embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of block-based calculation in Embodiment 1 of the present invention;

[0055] Figure 6 This is a schematic diagram of single-point calculation when the partition size is 1 data in Embodiment 2 of the present invention;

[0056] Figure 7 This is a schematic diagram of data preprocessing and calculation in Embodiment 4 of the present invention;

[0057] Figure 8 This is a schematic diagram of data preprocessing and calculation in Embodiment 6 of the present invention.

[0058] In the above figures, f1, f2, etc. represent individual feature data; w1, w2, etc. represent individual weight data; out1, out2, etc. are output result data;

[0059] MUL ROW stands for multiplier group, and MUL stands for multiplier;

[0060] Freg1, freg2, etc. represent the feature data register group; Wreg1-Wreg8, etc. represent the weight data register group;

[0061] Adder tree represents the adder tree, and count1, count2, etc. represent counters;

[0062] Figure 5 , Figure 6 In the diagram, ①-⑥ indicate the direction and order of data updates; Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] In one embodiment of the present invention, a convolution calculation device is provided, which can be used to process images, speech, or text. The hardware used in the embodiment is as follows: Figure 3 As shown, it specifically includes external memory, internal memory, and a computing unit. The computing unit is as follows... Figure 4 As shown, it includes a computing module, a storage module, and a control module (controller). The storage module is the register set.

[0065] The computing module contains multiple multiplier groups; in this embodiment, the number of multiplier groups is 8.

[0066] Each multiplier group contains multiple multipliers. In this embodiment, each multiplier group contains eight multipliers, from the first to the eighth (MUL1-MUL8).

[0067] The multiplier has a two-input, one-output structure. The two inputs are connected to a register set, one of which is connected to the feature data buffer, and the other is connected to the weight data buffer.

[0068] In this embodiment, the feature data cache includes eight feature data register groups (freg1-freg8) from the first to the eighth, and the weight data cache includes eight weight data register groups (Wreg1-Wreg8) from the first to the eighth. Each register group can store eight data items (feature data f1-f8, weight data w1-w8). Each multiplier is connected to three feature data register groups and three corresponding weight register groups.

[0069] The control module is connected to the multiplier and is used to control the input of the multiplier and the dynamic distribution of the input queue.

[0070] The multiple multiplier groups are connected to the adder tree to perform addition calculations on the multiplication results.

[0071] This invention provides a convolution calculation method, which implements convolution calculation using the aforementioned hardware. Specific embodiments are as follows:

[0072] Example 1:

[0073] In this embodiment, the weight data is sparse data, and the feature data is dense data.

[0074] First, the sparse weighted data is compressed, specifically including:

[0075] The weight data is expanded into row vectors and reassembled into a weight data matrix; the weight data is represented in the form of convolution kernels. The weight data in each convolution kernel is expanded into a row vector, which serves as a row in the weight data matrix, until the weight data of all convolution kernels in all channels are expanded into row vectors to form the weight data matrix. The weight data matrix contains the convolution kernel parameters at all positions in all channels of all convolution kernels.

[0076] The weight data matrix is ​​compressed using a compression algorithm, including but not limited to COO encoding and CSF encoding. The weight data matrix is ​​then compressed using either COO or CSF encoding and stored in external memory.

[0077] The data required for subsequent calculations is transferred from external memory to internal memory, and then transferred to multiple multiplier groups in the calculation module. The feature data is propagated by broadcast, and the feature data in each multiplier group is the same. The weight data is transmitted by distribution, and the weight data in each multiplier group is different.

[0078] The data is transmitted in blocks to the storage module of the computing module.

[0079] In this invention, "blocking" has two meanings: firstly, it is a verb-object phrase meaning to divide data into blocks; secondly, it is a noun specifically referring to a set of data after the block division. The distinction between the two meanings can be determined by their context, and no redundant distinction or limitation is made here.

[0080] In this embodiment, the method for segmenting the data to form the blocks is as follows (see...). Figure 5 The feature data and / or weight data are divided into multiple partitions on a two-dimensional level and into multiple channel groups on a channel level; one partition and one channel group of the feature data constitute a feature block, and one partition and one channel group of the weight data constitute a weight block.

[0081] by Figure 5 Taking the feature data diagram as an example, the input feature data can be represented as N×C×H×W, where N is the number of input feature data, C is the feature data channel, H is the height of the feature data matrix, and W is the width of the feature data matrix. In this embodiment, the first feature block input to the computing unit is shown in the dark part of the figure. The cross-section of the dark part is a partition of the feature data matrix (a partition on the two-dimensional level of feature data, with a size of 7×7). The extension marked Tc forms a channel group (containing 64 channels) on the channel level. The partition and channel group define a feature block, which contains a total of 7×7×64 feature data.

[0082] The rules for constructing the weight blocks are consistent with those for constructing the feature blocks. In this embodiment, the partition size of the weight data is 5×5, one channel group contains 64 channels, and one weight block contains a total of 5×5×64 weight data.

[0083] For a single feature block and weight block, after being input into the internal memory, the data of the block is input into the storage module of the computing unit in the order of the data groups. Then, the computing module performs multiplication calculations on the feature block and weight block. In this embodiment, the data group is 1×1×64 data, and each multiplier group is allocated 1×1×8 data. Different data groups are input into the computing unit storage module in the following order:

[0084] Within the partition, select all feature data and weight data (e.g., from left to right and top to bottom) at the same position within the channel group. Figure 5 The dark bar data marked with Tc (i.e., the data group) are input into the storage module in the computing unit's computing module, waiting for multiplication calculation.

[0085] After inputting all data within the same block into the storage module in the order described above, the blocks stored in the internal memory are updated as follows:

[0086] Within the same partition, feature blocks and corresponding weight blocks are input into the computing unit for calculation in channel order until all channels have been calculated. Then the partition is updated, and the above steps are repeated until all feature data have been calculated.

[0087] See attached document Figure 5 After the calculation of the dark blue blocks in the figure is completed, in the order indicated by ③, new feature blocks and weight blocks are extracted under the corresponding partitions according to the channel direction, and then input into the calculation module of the calculation unit in the above order.

[0088] For feature data, after all channels in the same partition have been calculated, the partitions are updated in order from left to right and from top to bottom. Channel groups are selected within the new partition range to form feature blocks. For weight data, since the partition size of the weight module is the same as that of the convolution kernel, it does not need to be updated. Instead, its initial state is restored, the first block is input into the calculation module of the calculation unit, and the above process is repeated until all feature data has been calculated.

[0089] The following is a description of the queues for dynamically allocating the inputs of the multiplier; see details below. Figure 4 .

[0090] The storage module of the computing unit includes internal memory and registers. The internal memory stores feature block data and weight block data. The registers are connected to the internal memory and the computing module and are used to cache data input to the computing module. The registers include multiple feature data registers and weight data registers. Furthermore, the registers are equipped with counters to count the number of remaining, uncalculated, valid data items in the registers.

[0091] In this embodiment, the convolution calculation device includes 8 feature data register groups and 8 weight data register groups, each of the 8 weight data register groups having a counter.

[0092] The internal memory stores feature data and weight data sequentially into register groups according to channel order. The first feature data register group and the first weight data register group store 8 data items for channels 1-8; the second feature data register group and the second weight data register group store 8 data items for channels 9-16, and so on. In this embodiment, the weight data is compressed data, and the weight data register group only stores valid data (i.e., non-zero value data), leaving the extra registers idle.

[0093] The steps for multiplication are as follows, with appendix attached. Figure 4Taking the second multiplier as an example, the second multiplier can obtain feature data from the first to third feature data register groups and weight data from the first to third weight data register groups. The following describes the method for dynamically allocating the multiplier input queue in this embodiment, using two calculation cycles as an example:

[0094] In the first multiplication calculation cycle: the second multiplier sequentially retrieves a valid weight data w9 from the second weight register group and transmits the channel position information of this weight data to the calculation unit control module. The control module uses this channel position information to retrieve the feature data f9 from the corresponding second feature register group for multiplication. The outputs of the eight multipliers are added in the adder tree, and the addition result is transmitted to the accumulator for accumulation.

[0095] In the second multiplication cycle: the multiplication of w9 and f9 data has been completed. At this point, there is no valid weight data in the second weight data register group that has not yet been calculated. The counter (count2) of the second weight data register group is 0, the counter (count1) of the first weight data register group is 2, and the counter (count3) of the third weight data register group is 3. Data is dynamically allocated according to the principle of adjacent queue priority to avoid the multiplier being idle: w20 data in the third weight data register group is transferred to the second multiplier, w2 data in the first weight data register group is transferred to the first multiplier, and w21 data in the third weight data register group is transferred to the third multiplier. The control module then transfers the f20, f2, and f21 data in the corresponding feature data register groups to the multipliers for multiplication and subsequent addition operations, respectively; subsequent calculations follow this pattern.

[0096] In this embodiment, the rules for data transfer between register groups (multiplier input queues) are as follows:

[0097] Let a register be numbered m. When there is no valid data in register set m, data is moved from register set m+1 first.

[0098] When the data in register m+1 is insufficient to move the data, move the data from register m-1.

[0099] To avoid data races, when the valid data in register group m-1 is 2 and there is no need to move data in register group m-2, data is moved from m-1.

[0100] The output and the result are first stored in internal registers, and then retrieved in subsequent calculations. They are then added to the multiplication result through an accumulator until the final accumulated result is calculated.

[0101] Example 2:

[0102] In this embodiment, the weight data is sparse data, and the feature data is dense data.

[0103] This embodiment is an improvement upon Embodiment 1. The main difference between this embodiment and Embodiment 1 is that the partitions constituting the feature blocks and weight blocks are a single data set (see details). Figure 6 The feature blocks and weight blocks are both 1×1×n (where n is the number of channels), and calculations are performed based on this to prioritize obtaining the final result of a single output data point. This calculation method can be called single-point calculation.

[0104] The descriptions of the remaining methods or apparatus are the same as in Example 1.

[0105] Example 3:

[0106] In this embodiment, the weight data is dense data, and the feature data is sparse data.

[0107] This embodiment is an optimization based on Embodiment 1 or 2 for different data. The main difference between this embodiment and Embodiment 1 is:

[0108] 1. In this embodiment, the sparse feature data will be compressed before calculation;

[0109] 2. In this embodiment, the feature data register group in the calculation unit is equipped with a counter to calculate the number of remaining valid data in the register that has not been calculated.

[0110] The descriptions of the remaining methods or apparatus are the same as in Example 1.

[0111] Example 4:

[0112] In this embodiment, both the weight data and the feature data are sparse data.

[0113] This embodiment is an optimization based on Embodiment 1 or 2 for different data. The main difference between this embodiment and Embodiment 1 is:

[0114] 1. Compress both the weight data and feature data before calculation;

[0115] 2. Simultaneously, the data input to the multiplier group of the computation unit consists of data from different channels. Due to data sparsity, there may be mismatches between weight data and feature data. Therefore, a preprocessing operation is performed on the data before computation. The preprocessing operation includes filtering out the weight data to be calculated based on the channel information of the feature data before inputting it into the weight register group to improve computational efficiency.

[0116] A simple example has been described, such as Figure 7As shown, the feature data and weight data are f1, f2, f8 and w1, w2, w3, respectively. Among them, only w1 (corresponding to f1) and w2 (corresponding to f2) are valid weight data and need to be multiplied. Since w3 has no data to be multiplied by f3, it can be regarded as invalid data and is removed during the preprocessing stage. Only the valid weight data is retained in the weight register group.

[0117] Example 5:

[0118] In this embodiment, both the weight data and the feature data are sparse data.

[0119] Example 5 is an improvement on Example 4. The main difference is that Example 5 preprocesses the feature data based on its sparse feature data characteristics, filters out the weight data to be calculated based on the channel information of the weight data, and then transmits the data to the feature register group. All other steps remain the same.

[0120] Example 6:

[0121] In this embodiment, both the weight data and the feature data are sparse data.

[0122] Example 6 is an improvement upon Example 4. The main difference is that Example 6 preprocesses both feature data and weight data based on their characteristics. Based on the channel information of the weight data, the feature and weight data to be calculated are selected, and then the data is transferred to the register set.

[0123] Here is an example, such as Figure 8 As shown, the feature data and weight data are f1, f2, f8 and w1, w2, w3, respectively. Among them, only f1 (corresponding to w1) and f2 (corresponding to w2) are feature data that need to be calculated. Since there is no w8 data for effective multiplication, f8 can be regarded as invalid data and is removed during the preprocessing stage. Only the data that needs to be calculated is retained in the feature register group and the weight register group.

[0124] Example 7:

[0125] In this embodiment, the weight data is sparse data, and the feature data is dense data.

[0126] This embodiment is an improvement on Embodiment 1, the main difference being: it adds a multiplier to Example 1 that skips the multiplication operation for zero values, specifically including:

[0127] When the controller finds that the feature data corresponding to the valid weight data is 0, the multiplier is turned off in this round of calculation, and no operation or power consumption is performed. By skipping invalid operations, that is, multiplication operations involving 0 values, the power consumption of the multiplier is reduced.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0129] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A convolution calculation method for image processing, characterized in that, Feature data and weight data are input to the computing unit in blocks via internal memory, and the multiplier input queue is dynamically allocated; the feature data is image feature data, and the weight data is obtained after training the neural network model. If the weight data is sparse, it is compressed; if the feature data is sparse, it is compressed; the feature data and / or weight data are divided into multiple partitions in a two-dimensional plane and into multiple channel groups in a channel plane; one partition and one channel group of the feature data constitute a feature block, and one partition and one channel group of the weight data constitute a weight block; for a single feature block and weight block, after being input into the internal memory, all feature data and weight data in the same position within the channel group are selected in a left-to-right and top-to-bottom order within the partition and input into the storage module of the computing module of the computing unit, and then the computing module performs multiplication calculation on the feature block and weight block; The dynamic allocation of multiplier input queues specifically includes: when the multiplier input queue is empty, data is moved from other multiplier input queues in sequence for multiplication calculation, with priority given to moving data from adjacent multiplier input queues.

2. The convolution calculation method according to claim 1, characterized in that... The internal memory stores the blocks sequentially into the queue at the input of the multiplier according to the channel order.

3. The convolution calculation method according to claim 1, characterized in that, The block input ensures that the outputs of a single multiplier group are coherent, and the input weight data of each multiplier in the same multiplier group belong to different channels of the same convolution kernel.

4. The convolution calculation method according to claim 1, characterized in that, Both the weight data and the feature data are sparse data.

5. The convolution calculation method according to claim 4, characterized in that, The feature data and weight data are preprocessed and filtered to select valid data before being input into the internal memory.

6. The convolution calculation method according to claim 1, characterized in that, The calculation unit skips invalid calculations based on the content of the input data.

7. The convolution calculation method according to claim 1, characterized in that, The method is used to process speech and text, the feature data is speech data and text data, and the weight data is obtained after the speech and text are trained by a neural network model.

8. A convolution calculation device, comprising: External memory, internal memory, and computing units; The feature data and weight data in the external memory are input into the internal memory in blocks; The computing unit includes a computing module, a storage module, and a control module; The computation module includes multiple multiplier groups and an adder tree, wherein the multiplier groups include multiple multipliers; The storage module is connected to the internal memory and the computing module, and is used to cache the input data of the computing module; The control module connects the computing module and the storage module, and is used to control the input of the multiplier and the dynamic distribution of the input queue; If the weight data is sparse, it is compressed; if the feature data is sparse, it is compressed; the feature data and / or weight data are divided into multiple partitions in a two-dimensional plane and into multiple channel groups in a channel plane; one partition and one channel group of the feature data constitute a feature block, and one partition and one channel group of the weight data constitute a weight block; for a single feature block and weight block, after being input into the internal memory, all feature data and weight data in the same position within the channel group are selected in a left-to-right and top-to-bottom order within the partition and input into the storage module of the computing module of the computing unit, and then the computing module performs multiplication calculation on the feature block and weight block; The dynamic allocation of multiplier input queues specifically includes: when the multiplier input queue is empty, data is moved from other multiplier input queues in sequence for multiplication calculation, with priority given to moving data from adjacent multiplier input queues.

Images