Subflow type sparse convolution rule table generation method, convolution operation method and device

Abstract

The invention provides a generation method of a sub-flow type sparse convolution rule table and a convolution operation method and device, and provides the generation method of the sub-flow type sparse convolution rule table for realizing sub-flow type sparse convolution on a tensor processing unit, and the generation method comprises the steps: calculating the offset of an input coordinate; calculating corresponding output coordinates of the input coordinates in each filter kernel, and calculating the offset of the output coordinates; the offset of the input coordinates and the offset of the output coordinates corresponding to the filter core are compared, a sequence mask corresponding to the filter core is obtained, and elements, not equal to 0, in the sequence mask indicate that the offset of the corresponding set of input coordinates is the same as the offset of the corresponding set of output coordinates; and obtaining a rule table corresponding to the filter core based on the sequence mask corresponding to the filter core and a mask screening function, and performing substream sparse convolution operation based on the rule table. The number of times of data access is effectively reduced, and the operation efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method for generating a sub-flow type sparse convolution rule table, a convolution operation method and a device. Background Art

[0002] Detection algorithms based on point cloud data, since point cloud data is a typical sparse data, in deep learning algorithms based on point cloud data, sparse convolution (Spconv) operators are often used for convolution operations. The sparse convolution operator establishes the mapping relationship between the activation input data (non-zero data) and the output data index through the generated rule table, which greatly reduces the amount of calculation and memory access.

[0003] Common sparse convolution operators are implemented based on GPU (Garphics Processing Unit), which establishes a hash table of input-output tensor numbers, and then establishes a mapping relationship between input numbers and output numbers in the hash table based on the rule table. TPU (Tensor Processing Unit) is cheaper than GPU, and TPU is more suitable for neural network reasoning. In some scenarios, users choose to perform sparse convolution operations based on TPU. Due to the differences in TPU and GPU architecture and performance, the solution for generating rule tables in GPU cannot be directly transplanted to TPU.

[0004] Therefore, there is an urgent need to provide a solution for implementing sparse convolution operators on TPU. Summary of the invention

[0005] The present application provides a method for generating a sub-stream sparse convolution rule table, a convolution operation method and a device, and implements a sub-stream sparse convolution operator on a TPU.

[0006] In a first aspect, the present application provides a method for generating a sub-flow type sparse convolution rule table, which is applied to a tensor processing unit, and the method comprises: Calculate the offset of the input coordinates; Calculating the output coordinates corresponding to the input coordinates in each filter kernel, and calculating the offset of the output coordinates; Compare the offset of the input coordinates and the offset of the output coordinates corresponding to the filter kernel to obtain a sequence mask corresponding to the filter kernel, wherein an element in the sequence mask that is not 0 indicates that the offset of a corresponding set of input coordinates is the same as the offset of the output coordinates; Based on the sequence mask corresponding to the filter kernel and the mask screening function, a rule table corresponding to the filter kernel is obtained, so as to perform a sub-flow type sparse convolution operation based on the rule table.

[0007] In a second aspect, the present application provides a sub-flow type sparse convolution operation method, applied to a tensor processing unit, comprising: Based on the index of the input coordinates in the rule table corresponding to each filter core, the activation input data is extracted from the input data and aggregated to obtain the activation input matrix corresponding to each filter core; Multiply the activation input matrix corresponding to each filter core by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; Based on the rule table corresponding to each filter core, the corresponding activation output data of each filter core is scattered to the corresponding output coordinate, and the output data under the same output coordinate is superimposed to obtain the operation result of sub-flow type sparse convolution; Among them, the rule table is generated based on the method for generating the sub-flow type sparse convolution rule table provided in the first aspect of the present application.

[0008] In a third aspect, the present application provides a device for generating a sub-flow type sparse convolution rule table, which is applied to a tensor processing unit, and the device includes: A first offset calculation module, used to calculate the offset of the input coordinates; A second offset calculation module, used to calculate the output coordinates corresponding to the input coordinates in each filter kernel, and calculate the offset of the output coordinates; An offset comparison module, used for comparing the offset of the input coordinates and the offset of the output coordinates corresponding to the filter kernel, to obtain a sequence mask corresponding to the filter kernel, wherein the elements in the sequence mask that are not 0 indicate that the offset of the corresponding set of input coordinates is the same as the offset of the output coordinates; A rule table generating module is used to determine a rule table corresponding to the filter kernel based on a sequence mask corresponding to the filter kernel, so as to perform a sub-flow type sparse convolution operation based on the rule table.

[0009] In a fourth aspect, the present application provides a tensor processing unit for performing sub-flow type sparse convolution operations, on which a sparse convolution rule table is stored, and the rule table is generated based on the method for generating the sub-flow type sparse convolution rule table provided in the first aspect of the present application.

[0010] In a fifth aspect, the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a tensor processing unit, they are used to implement the method provided in the first aspect or the second aspect of the present application.

[0011] In a sixth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a tensor processing unit, implements the method provided in the first aspect or the second aspect of the present application.

[0012] The sub-stream sparse convolution rule table generation method, convolution operation method and device provided in this application realize sub-stream sparse convolution operation on TPU. In the neural network application scenario, since TPU has a systolic array structure, it has better computing performance than GPU, which improves the efficiency of convolution operation. In order to implement the sub-manifold sparse convolution operation on the TPU, it is necessary to first calculate the offset between the input coordinates and the output coordinates of each filter core in the sub-manifold sparse convolution filter. By comparing the offsets of the input coordinates and the output coordinates, a mask matrix is ​​obtained, which is recorded as a sequence mask. The sequence mask is a 0-1 mask, in which the offsets of the input coordinates and the output coordinates corresponding to the elements with a value of 0 are different. Since the sub-manifold sparse convolution operation has the feature that other filter cores will be activated only when the input data covered by the filter (or weight) at the center of the kernel is a non-zero value, it can be analyzed that when the input data corresponding to the input coordinates is activated in a certain filter core, the corresponding output coordinates should be a certain input coordinate, so that through the mask screening operation with the sequence mask as the mask, the corresponding relationship between the input coordinates and the output coordinates corresponding to each filter core can be obtained, and then the mapping relationship between the input coordinates and the output coordinate index is established based on the corresponding relationship, and the rule table corresponding to each filter core is obtained, so as to implement the sub-manifold sparse convolution operation based on the rule table. By using mask and mask screening functions, sub-stream sparse convolution operations are implemented on TPU, which effectively reduces the number of data memory accesses. It is more suitable for implementing sub-stream sparse convolution operations on TPU and improves computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0014] Figure 1 A schematic diagram of a point cloud data processing flow provided in an embodiment of the present application; Figure 2 A schematic diagram of a flow chart of a method for generating a sub-flow type sparse convolution rule table provided in one embodiment of the present application; Figure 3 A schematic diagram of a filter offset vector provided by one embodiment of the present application; Figure 4 A schematic diagram of a sequence mask calculation process provided by an embodiment of the present application; Figure 5 A schematic diagram of a flow chart of a method for generating a sub-flow type sparse convolution rule table provided in another embodiment of the present application; Figure 6 A schematic diagram of a flow chart of a method for generating a sub-flow type sparse convolution rule table provided in yet another embodiment of the present application; Figure 7 A schematic diagram of a substream type sparse convolution operation process provided by an embodiment of the present application; Figure 8 A schematic diagram of a device for generating a sub-flow type sparse convolution rule table provided in one embodiment of the present application.

[0015] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0016] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0017] First, the terms involved in this application are explained: Submanifold Sparse Convolution (Subm-Spconv): also known as semi-manifold sparse convolution, is a type of sparse convolution. The output of submanifold sparse convolution is activated only when the input data covered by the center of the filter kernel is activated. The positions where the input data and output data of submanifold sparse convolution are not empty (or 0) are the same.

[0018] Mask: A matrix used to recalculate the values ​​of each element of a matrix, such as a 0-1 mask. The values ​​of the elements in the 0-1 mask are 0 or 1. By multiplying the 0-1 mask with the corresponding matrix, the elements in the matrix can be selected.

[0019] Mask filtering function: A function of the tensor processing unit that takes an array and a mask as input and extracts the element values ​​corresponding to the non-zero positions in the mask according to the index of these indexes in the input array.

[0020] Data aggregation function: input an index sequence and an array, based on the index in the index sequence, take out the values ​​from the corresponding index of the array and aggregate them together.

[0021] Point cloud data is widely used in many fields, such as target detection, 3D target reconstruction, and autonomous driving. Point cloud data is a typical sparse data. When processing point cloud data based on convolutional neural networks, since point cloud data contains a lot of data with values ​​of 0, in order to reduce the number of channels during convolution calculation, only non-zero values ​​in point cloud data are convolved, and sparse convolution is proposed. Submanifold sparse convolution is a common coefficient convolution.

[0022] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0023] Figure 1 A schematic diagram of a point cloud data processing flow provided in an embodiment of the present application is shown in FIG. Figure 1 As shown in the figure, taking the 3D target detection application scenario as an example, the point cloud data processing flow includes: collecting point cloud data, such as collecting point cloud data through lidar, 3D camera, etc.; preprocessing the point cloud data, such as noise reduction, registration, segmentation, etc.; inputting the preprocessed point cloud data into the convolutional neural network model for feature extraction and analysis, identifying the 3D target in the point cloud data, and realizing 3D target detection.

[0024] Convolutional neural networks for point cloud data usually use sparse convolution operators for convolution operations. Currently, the sparse convolution operator is only available on the GPU.

[0025] Compared with GPU, TPU has better computing performance due to its systolic array structure, which can improve the efficiency of convolution operations. Therefore, it is necessary to provide a solution for implementing sparse convolution operators for point cloud data on TPU.

[0026] The present application is directed to sub-manifold sparse convolution, which has an important characteristic, that is, when the input data covered by the center of the filter kernel is a non-zero value, other filter kernels will be activated for calculation to obtain output data. Combining the calculation relationship between the output coordinates and the input coordinates, it can be obtained that when the input coordinates of the sub-manifold sparse convolution are paired with a certain filter kernel (the eigenvalue at the input coordinates is a non-zero value), the corresponding output coordinates are a certain input coordinates, then the input coordinates are activated. Based on the above characteristics of sub-manifold sparse convolution, an embodiment of the present application provides a method for generating a mask-based sub-manifold sparse convolution rule table to implement sub-manifold sparse convolution on a TPU.

[0027] The operation process of sub-stream sparse convolution is divided into two steps. The first step is to generate a rule table (Rule Book). The second step is to aggregate the input data required by each filter core based on the generated rule table, and input the aggregated input data into the corresponding filter core to obtain the output data. Based on the output index in the rule table, the output data is dispersed to the corresponding position and the output data of different filters at this position are superimposed to realize the operation of sub-stream sparse convolution.

[0028] In order to adapt to the computing characteristics and memory access characteristics of TPU, the embodiment of the present application provides a method for generating a mask-based sub-stream sparse convolution rule table, which obtains a sequence mask by comparing the input coordinates with the offset of the input coordinates, and generates the above rule table based on the sequence mask and the mask screening function of TPU. Through the mask calculation method, the memory access operations required for rule table generation are reduced, the efficiency of rule table generation is improved, and thus the computational efficiency of sub-stream sparse convolution based on TPU is improved.

[0029] Figure 2 A schematic diagram of a flow chart of a method for generating a sub-stream type sparse convolution rule table provided in an embodiment of the present application, wherein the method for generating a sub-stream type sparse convolution rule table is applied to a tensor processing unit, such as Figure 2 As shown, the method for generating the sub-flow type sparse convolution rule table includes the following steps: Step S201, calculating the offset of the input coordinates.

[0030] The input coordinates are the coordinates of the input data, such as the coordinates of the input data in the input tensor, which can be three-dimensional coordinates. Specifically, they can be the coordinates of the input data with non-zero values.

[0031] Different input coordinates have different offsets, and the offset value can be continuous, such as an arithmetic progression starting from 0 with a tolerance of 1. The offset is used to represent the position offset relationship between different input coordinates.

[0032] Specifically, the offset of each input coordinate may be generated based on the number of input coordinates.

[0033] The input coordinates can be k-dimensional coordinates, where k can be 2, 3, 4 or other values. The input coordinates are [a1, a2, ,ak] as an example, ai is a natural number in the interval [0, ni], i is a positive integer from 1 to k, and ni is a positive integer. Then the input coordinates [a1, a2, , ak] The corresponding offset can be: , j is a positive integer greater than 1 and less than k.

[0034] Taking the four-dimensional coordinates of [a, b, c, d] as an example, where a, b, c and d are natural numbers in the intervals [0, n1], [0, n2], [0, n3] and [0, n4] respectively, the offset corresponding to the input coordinates [a, b, c, d] can be: a×n2×n3×n4+b×n3×n4+c×n4+d.

[0035] For example, the input coordinates are three-dimensional coordinates, and the dimension of the input coordinates is [41, 1600, 1408]. [41,1600, 1408] means that the input coordinates are divided into 41 parts in the z-axis direction, 1600 parts in the y-axis direction, and 1408 parts in the x-axis direction. Then, the offset of the input coordinates (0, 1, 2) is: 0×1600×1408+1×1408+2×1=1410.

[0036] Specifically, each input coordinate can be evenly stored in multiple local memories of the TPU, such as 64 local memories; the offset of the input coordinate can be calculated in parallel in the independent computing unit corresponding to each local memory. Specifically, the offset of the input coordinate can be obtained by calling the INT32 matrix multiplication instruction, multiplying the vector corresponding to the input coordinate by a preset vector on the left.

[0037] After the offsets of the input coordinates of all input data are calculated, the offsets of the input coordinates of the input data with non-zero values ​​may be extracted therefrom, so as to perform subsequent calculations based on the offsets of the input coordinates of the input data with non-zero values.

[0038] Step S202, calculating the output coordinates corresponding to the input coordinates in each filter kernel, and calculating the offset of the output coordinates.

[0039] The output coordinates are the coordinates of the output data, specifically the coordinates of the output data in the output tensor, which can be three-dimensional coordinates.

[0040] The calculation formula of the output coordinate is: Output coordinate = Input coordinate + Supplement vector - Filter offset vector. Among them, the filter offset vector and the supplement vector are both constant values. The value of the supplement vector in each dimension (such as x, y and z axis) is a fixed value, such as 1, and the filter offset vector is Figure 3 shown.

[0041] Figure 3 A schematic diagram of a filter offset vector provided by an embodiment of the present application, such as Figure 3 As shown, Figure 3 Taking a 3*3*3 filter as an example, the filter offset vector is marked on each filter kernel, such as (0, 0, 0), (0, 1, 0), etc.

[0042] The constant vector corresponding to each filter kernel can be calculated in advance, where the constant vector = supplementary vector - filter offset vector, and then the sum of the input coordinates and the constant vectors corresponding to each filter kernel is calculated to obtain the output coordinates corresponding to the input coordinates under each filter kernel.

[0043] The input coordinates are stored in multiple local memories of the TPU. The sum of the input coordinates stored therein and the constant vectors corresponding to each filter kernel can be calculated in parallel based on the addition calculation units of each local memory to obtain the corresponding output coordinates.

[0044] For example, taking the input coordinates as (0, 2, 1), the filter offset vector of the filter kernel is (1, 2, 0), and the corresponding output coordinates are (0, 1, 1).

[0045] The calculation method of the offset of the output coordinates can refer to the calculation method of the offset of the input coordinates. It only needs to replace the object from the input coordinates to the output coordinates, which will not be repeated here.

[0046] Step S203 , comparing the offset of the input coordinates and the offset of the output coordinates corresponding to the filter kernel to obtain a sequence mask corresponding to the filter kernel.

[0047] The elements in the sequence mask that are not 0 indicate that the offset of the corresponding set of input coordinates is the same as the offset of the output coordinates.

[0048] After obtaining the offset of each input coordinate and the offset of the output coordinate corresponding to each filter kernel, for some or all filter kernels in the filter of the network, the offset composition of the input coordinate is compared with the offset of the output coordinate corresponding to the filter kernel, and based on the comparison result, the sequence mask corresponding to the filter kernel is obtained.

[0049] Since the filter of the sub-flow type sparse convolution has the characteristic of symmetric activation, in order to save computing resources and improve the efficiency of rule table generation, the rule table calculation can be performed only on the first half of the filter kernel, and the rule table corresponding to the second half of the filter kernel is obtained based on the rule table corresponding to the symmetric filter kernel. The input coordinates of the middle filter kernel are equal to the output coordinates, and the index of the input coordinates and the output coordinates in its rule table are the same.

[0050] For example, Figure 3Taking the filter shown as an example, which includes 3*3*3 filter cores, the filter cores in the first half are the 1st to 13th filter cores, the filter cores in the second half are the 15th to 27th filter cores, and the middle filter core is the 14th filter core. Among them, the 1st filter core and the 27th filter core are symmetrically activated filter cores, the 2nd filter core and the 26th filter core are symmetrically activated filter cores, and so on.

[0051] Since the input coordinates and the input coordinate offsets are calculated in the same way, equal offsets indicate that a corresponding set of input coordinates are equal to the output coordinates. Therefore, by comparing whether the offsets are equal, it is possible to determine whether the output coordinates are equal to the input coordinates, thereby determining whether the corresponding input coordinates are activated.

[0052] Optionally, comparing the offset of the input coordinates and the offset of the output coordinates corresponding to the filter kernel to obtain a sequence mask corresponding to the filter kernel includes: The offset of the input coordinates and the offset of the output coordinates corresponding to the filter kernel are compared, and a first mask matrix is ​​obtained based on the comparison result; the first mask matrix is ​​left-multiplied by a preset sequence to obtain a sequence mask corresponding to the filter kernel.

[0053] Specifically, the offset of the input coordinate is taken as the column vector, and the offset of the output coordinate corresponding to the filter kernel is taken as the column vector, and comparison is performed element by element. If the i-th offset in the column vector is the same as the j-th offset in the row vector, then the element in the i-th row and j-th column of the first mask matrix is ​​not 0 (for example, it is 1), otherwise it is 0.

[0054] For example, Figure 4 A schematic diagram of a sequence mask calculation process provided for an embodiment of the present application is shown in FIG. Figure 4 As shown in FIG. 1 , the offset of the input coordinate of the non-zero input data is [5, 6, 11, 14, 16], and the offset of the output coordinate corresponding to the input coordinate of the non-zero input data under one filter kernel is [5, 7, 9, 11, 16]. Then, the offset of the input coordinate is taken as the column vector, and the offset of the output coordinate is taken as the row vector. By comparing the elements one by one, the first mask matrix (0-1 mask matrix) is obtained, and 0-1 indicates that the element value is 0 or 1. Then, the first mask matrix is ​​multiplied by a preset sequence on the left, and the preset sequence is an arithmetic sequence, such as an arithmetic sequence starting from 1 and with a common difference of 1, to obtain the sequence mask under the corresponding filter kernel, that is, [1, 0, 0, 3, 5]. The output coordinate at the position of the non-zero element in the sequence mask is one of the input coordinates (that is, the offset is the same), that is, the output coordinate corresponding to the 1st, 4th, and 5th offsets has the same offset as the input coordinate.

[0055] Step S204: Based on the sequence mask corresponding to the filter kernel and the mask screening function, a rule table corresponding to the filter kernel is obtained, so as to perform a sub-manifold sparse convolution operation based on the rule table.

[0056] After calculating the sequence mask corresponding to the filter kernel, based on the mask screening function of TPU, the sequence mask is used as the mask to obtain the mapping relationship between the index of the input coordinates of the filter kernel and the index of the output coordinates, that is, to obtain the rule table corresponding to the filter kernel, and so on.

[0057] Since the submanifold sparse convolution has the characteristic of symmetric activation, for the two filter kernels symmetrically activated in the network, it is only necessary to obtain the sequence mask of one of the filter kernels based on calculation, and based on the sequence mask and the mask screening function, obtain the rule table corresponding to the filter kernel, and the rule table corresponding to the other filter kernel can be obtained based on the rule table corresponding to the filter kernel symmetrically activated with the filter kernel.

[0058] For two symmetrically activated filter cores, such as filter core A and filter core B, the index of the input coordinate of the rule table of filter core B is the index of the output coordinate of the rule table of filter core A, and the index of the output coordinate of the rule table of filter core B is the index of the input coordinate of the rule table of filter core A.

[0059] After obtaining the rule table corresponding to each filter kernel in the network, a submanifold convolution operation is performed based on each rule table. Specifically, first, based on the index of the input coordinates in the rule table corresponding to the filter kernel, the input data corresponding to the filter kernel is extracted from all input data (including zero values ​​and non-zero values) and aggregated together, and then the aggregated input data is matrix multiplied with the filter kernel to obtain the output data of the filter kernel. Based on the index of the output coordinates in the rule table corresponding to the filter kernel, the output data is dispersed to the corresponding position; when different filter kernels have output data at the same output coordinate, the output data of the same output coordinate of different filter kernels are superimposed to obtain the operation result of the submanifold sparse convolution, that is, the output tensor.

[0060] The method for generating a sub-stream sparse convolution rule table provided in this embodiment implements sub-stream sparse convolution operations on the TPU. In the neural network application scenario, since the TPU has a systolic array structure, it has better computing performance than the GPU, thereby improving the efficiency of the convolution operation. In order to implement the sub-manifold sparse convolution operation on the TPU, it is necessary to first calculate the offset between the input coordinates and the output coordinates of each filter core in the sub-manifold sparse convolution filter. By comparing the offsets of the input coordinates and the output coordinates, a mask matrix is ​​obtained, which is recorded as a sequence mask. The sequence mask is a 0-1 mask, in which the offsets of the input coordinates and the output coordinates corresponding to the elements with a value of 0 are different. Since the sub-manifold sparse convolution operation has the feature that other filter cores will be activated only when the input data covered by the filter (or weight) at the center of the kernel is a non-zero value, it can be analyzed that when the input data corresponding to the input coordinates is activated in a certain filter core, the corresponding output coordinates should be a certain input coordinate, so that through the mask screening operation with the sequence mask as the mask, the corresponding relationship between the input coordinates and the output coordinates corresponding to each filter core can be obtained, and then the mapping relationship between the input coordinates and the output coordinate index is established based on the corresponding relationship, and the rule table corresponding to each filter core is obtained, so as to implement the sub-manifold sparse convolution operation based on the rule table. By using mask and mask screening functions, sub-stream sparse convolution operations are implemented on TPU, which effectively reduces the number of data memory accesses. It is more suitable for implementing sub-stream sparse convolution operations on TPU and improves computing efficiency.

[0061] Figure 5 A flow chart of a method for generating a substream type sparse convolution rule table provided in another embodiment of the present application. Figure 2 Based on the embodiment shown, step S204 is further refined, and a step of performing a sub-flow type sparse convolution operation based on the generated rule table is added after step S204, such as Figure 5 As shown, the method for generating a sub-flow type sparse convolution rule table provided in this embodiment may include the following steps: Step S501, calculating the offset of the input coordinates.

[0062] Step S502, calculating the output coordinates corresponding to the input coordinates in each filter kernel, and calculating the offset of the output coordinates.

[0063] Optionally, calculate the offset of the input coordinates or output coordinates, including: Based on the dimension of the input coordinate or the output coordinate, an offset vector is constructed; and the transpose of the offset vector is multiplied on the left by the vector corresponding to the input coordinate or the vector corresponding to the output coordinate to obtain the offset of the input coordinate or the output coordinate.

[0064] Step S503 , comparing the offset of the input coordinates and the offset of the output coordinates corresponding to the filter kernel, obtaining a first mask matrix based on the comparison result, and multiplying the first mask matrix by a preset sequence to obtain a sequence mask corresponding to the filter kernel.

[0065] The elements in the sequence mask that are not 0 indicate that the offset of the corresponding set of input coordinates is the same as the offset of the output coordinates.

[0066] Step S504 : Taking the sequence mask corresponding to the filter kernel as input and as a mask, based on the mask screening function, determine the index of the output coordinate corresponding to the filter kernel.

[0067] The sequence mask corresponding to the filter kernel is used as both the mask and the input, that is, the sequence mask corresponding to the filter kernel is the input of the mask screening and the sequence mask corresponding to the filter kernel is the mask of the mask screening. Based on the mask screening function, the value of the non-zero position index of the mask in the input is extracted as the index of the corresponding output coordinate.

[0068] Since the index starts from 0, we can subtract 1 from the index output by the mask filtering function to get the final index of the output coordinates.

[0069] by Figure 4 Taking the sequence mask in the illustrated embodiment as an example, with the sequence mask [1, 0, 0, 4, 5] as the mask and input, based on the mask screening function, the indexes of the extracted output coordinates are 1, 4 and 5 respectively, and after subtracting 1, the final index ratios are 0, 3 and 4.

[0070] Step S505: Generate an index sequence based on the number of offsets of the input coordinates, where the index sequence is an arithmetic progression starting from 0 and having a step length of 1.

[0071] The index sequence includes multiple indexes, and the number of indexes is equal to the number of offsets of the input coordinates of all input data (including input data with values ​​of 0 and non-zero).

[0072] Exemplarily, taking 15470 input coordinates as an example, the index sequence starts from 0, has a step length of 1, and is an arithmetic progression of 15470 numbers in total.

[0073] Step S506 : using the sequence mask corresponding to the filter kernel as a mask and the index sequence as input, determining the index of the input coordinate corresponding to the filter kernel.

[0074] Based on the sequence mask, the value at the non-zero position in the sequence mask is extracted from the index sequence to obtain the index of the input coordinate.

[0075] Step S507 , establishing a mapping relationship between the index of the output coordinates corresponding to the filter kernel and the index of the input coordinates, and obtaining a rule table corresponding to the filter kernel.

[0076] For part or all of the filter kernels, based on the correspondence between the input coordinates and the output coordinates, a mapping relationship between the index of the input coordinates corresponding to the filter kernel and the index of the output coordinates is established to obtain a rule table corresponding to the filter kernel.

[0077] The rule table includes two columns, one of which is the index of the input coordinates and the other is the index of the corresponding output coordinates.

[0078] Due to the symmetric activation characteristics of the submanifold sparse convolution, the aforementioned method can be used to generate the rule table only for the first half, i.e., the first (N-1) / 2 filter cores, where N is the total number of filter cores in the network and N is an odd number. For the second half, i.e., the (N+3) / 2th to the Nth filter cores, the rule table corresponding to the filter core is generated based on the rule table corresponding to the symmetric activated filter core through the symmetric activation feature. For the middle filter core, i.e., the (N+1) / 2th filter core, since all the input data with values ​​are activated and the input coordinates are equal to the output coordinates, the index of the input coordinates in its rule table is the same as the index of the output coordinates.

[0079] The 1st to (N-1) / 2nd filter cores can be traversed, and the method provided in the above embodiment can be used for each filter core to obtain the rule table corresponding to the filter core. For the (N+3) / 2nd to Nth filter cores, the rule table corresponding to the filter core is generated based on the rule table corresponding to the symmetrically activated filter core by symmetrically activating the feature. The index of the input coordinate in the rule table corresponding to the (N+1) / 2nd filter core is the same as the index of the output coordinate. The index of the input coordinate in the rule table corresponding to the (N+1) / 2nd filter core can be determined based on the offset of the input coordinate of the non-zero input data.

[0080] After obtaining the rule table corresponding to each filter core in the network, the method for generating the sub-flow type sparse convolution rule table provided in this embodiment may further include the step of performing a sub-flow type sparse convolution operation based on the generated rule table, namely, step S508 to step S510.

[0081] Step S508 , based on the index of the input coordinates in the rule table corresponding to each filter core, extract the activated input data from the input data and aggregate them to obtain the activated input matrix corresponding to each filter core.

[0082] During the submanifold sparse convolution operation, for each filter kernel, based on the index of the input coordinates in the rule table corresponding to the filter, the activation input data of the filter kernel is extracted from the input data (such as the input tensor) and aggregated together to form the activation input matrix corresponding to the filter kernel.

[0083] The activation input data is a non-zero value and is the input data corresponding to the index of the input coordinate in the rule table.

[0084] Specifically, based on the data aggregation function of TPU, the input tensor can be used as input, the index of the input coordinates in the rule table can be used as the index sequence, the input data (activated input data) corresponding to the index of the input coordinates can be taken out from the input tensor and aggregated together to obtain the activated input matrix.

[0085] Step S509, multiplying the activation input matrix corresponding to each filter core by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core.

[0086] Step S510, based on the rule table corresponding to each filter core, scatter the corresponding activated output data of each filter core to the corresponding output coordinate, and superimpose the output data under the same output coordinate to obtain the operation result of the sub-flow type sparse convolution.

[0087] After obtaining multiple activation output data of the filter core, based on the index of the output coordinate corresponding to each activation data in the rule table, the multiple activation output data are scattered or dispersed to the corresponding output coordinate. The output coordinate corresponding to the activation output data is the output coordinate corresponding to the input coordinate of the input data of the activation output data calculated. When the activation output data of different filter cores correspond to the index of the same output coordinate, the activation output data of different filters are superimposed to obtain the final activation output data at the output coordinate. After the operation is completed, the operation result of the sub-flow type sparse convolution, that is, the output tensor, is obtained.

[0088] Furthermore, the output tensor can also be sent to a downstream device or module for further analysis, such as target recognition based on the output tensor.

[0089] In this embodiment, a mask sequence is obtained by comparing the offsets, and the indexes of the input coordinates and the output coordinates are obtained based on the mask sequence and the mask screening function, and a mapping relationship is established to obtain a rule table. This does not require excessive data memory access, and the calculation operation is simple, which can better adapt to the calculation characteristics and memory access characteristics of the TPU, and improves the efficiency of the sub-stream sparse convolution operation implemented on the TPU. After obtaining the rule table, the operation result of the sub-stream sparse convolution is obtained by input data aggregation, output data scattering and superposition, and the sub-stream sparse convolution operation is implemented on the TPU, which improves the efficiency of the sub-stream sparse convolution operation and reduces the calculation cost.

[0090] Figure 6 A flowchart of a method for generating a substream type sparse convolution rule table provided in another embodiment of the present application is provided in this embodiment. Figure 2 On the basis of the illustrated embodiment, the step of calculating the offset and step S204 are further refined, and the filter kernel of the sub-flow type sparse convolution includes multiple groups of symmetrically activated first filter kernels and second filter kernels, and an intermediate filter kernel.

[0091] like Figure 6 As shown, the method for generating a sub-flow type sparse convolution rule table provided in this embodiment may include the following steps: Step S601, calculating the output coordinates corresponding to the input coordinates in each filter kernel.

[0092] Step S602, for the input coordinates and the output coordinates, the offset is calculated in the following manner: based on the dimension of the coordinates, an offset vector is constructed, and the offset vector is transposed and the vector corresponding to the coordinate is left-handed to obtain the offset of the coordinates.

[0093] The dimensions of a coordinate (including input coordinates and output coordinates) are the number of times the coordinate is divided on the x, y, and z axes.

[0094] When the input coordinates and output coordinates are three-dimensional coordinates, the offset vector includes 3 elements. Taking the coordinate dimension [z, y, x] as an example, the first element of the offset vector is the product of y and x, the second element is x, and the third element is 1.

[0095] The vector corresponding to the coordinate is the value of each dimension in the coordinate as a value in the vector. Taking the coordinate (a, b, c) as an example, the corresponding vector, taking the row vector as an example, is [a, b, c].

[0096] For example, take the coordinate dimension [41, 1600, 1408] as an example. [41, 1600, 1408] means that the input coordinate is divided into 41 parts in the z-axis direction, 1600 parts in the y-axis direction, and 1408 parts in the x-axis direction. The offset vector is [1600*1408, 1408, 1], where "*" represents the multiplication sign. The vector corresponding to the coordinate (0, 1, 2) is [0, 1,2]. The transpose of [1600*1408, 1408, 1] is multiplied by [0, 1, 2] to obtain an offset of 1410.

[0097] Step S603 : for each first filter kernel, compare the offset of the input coordinates and the offset of the output coordinates corresponding to the first filter kernel to obtain a sequence mask corresponding to the first filter kernel.

[0098] Optionally, comparing the offset of the input coordinates and the offset of the output coordinates corresponding to the first filter kernel to obtain a sequence mask corresponding to the first filter kernel includes: Compare the offset of the input coordinates and the offset of the output coordinates corresponding to the first filter kernel, and obtain a first mask matrix based on the comparison result; multiply the first mask matrix by a preset sequence to obtain a sequence mask corresponding to the first filter kernel.

[0099] Step S604: determining a rule table corresponding to the first filter kernel based on the sequence mask and the mask screening function corresponding to the first filter kernel.

[0100] The rule table corresponding to the first filter kernel can be generated based on the method provided by any of the aforementioned embodiments. For example, the sequence mask corresponding to the first filter kernel is used as input and mask, and based on the mask screening function, the index of the output coordinate corresponding to the first filter kernel is determined; the sequence mask corresponding to the first filter kernel is used as mask, and the index sequence is used as input to determine the index of the input coordinate corresponding to the first filter kernel; a mapping relationship between the index of the output coordinate corresponding to the first filter kernel and the index of the input coordinate is established to obtain the rule table corresponding to the first filter kernel.

[0101] Step S605 : for each second filter core, based on the rule table corresponding to the first filter core symmetrically activated with the second filter core, generate a rule table corresponding to the second filter core.

[0102] The rule table corresponding to the second filter core that is symmetrically activated with the first filter core can be obtained by exchanging the index of the input coordinates with the index of the output coordinates in the rule table corresponding to the first filter core.

[0103] After obtaining a rule table corresponding to a first filter core, the rule table corresponding to the first filter core can be immediately obtained based on the rule table corresponding to the first filter core. Alternatively, after all the rule tables corresponding to the first filter cores are generated, the rule tables corresponding to the second filter cores can be generated.

[0104] Optionally, generating a rule table corresponding to the second filter kernel based on a rule table corresponding to a first filter kernel that is symmetric to the second filter kernel includes: Taking the index of the input coordinate in the rule table corresponding to the first filter kernel as the index of the output coordinate, and taking the index of the output coordinate in the rule table corresponding to the first filter kernel as the index of the input coordinate, a mapping relationship between the index of the output coordinate and the index of the input coordinate corresponding to the second filter kernel is established to obtain the rule table corresponding to the filter of the second filter kernel.

[0105] Step S606: determining a rule table corresponding to the intermediate filter kernel based on the offset of the input coordinates.

[0106] Since in the intermediate filter kernel, the input coordinates are equal to the output coordinates, and all non-zero input data are activated, the index of the input coordinates can be obtained based on the offset of the input coordinates of the non-zero input data. Since the input coordinates are equal to the output coordinates, the index of the input coordinates in the rule table corresponding to the intermediate filter kernel is also the same as the index of the output coordinates.

[0107] Step S607, performing a sub-flow type sparse convolution operation based on the rule table corresponding to each filter kernel to obtain an output tensor.

[0108] Specifically, based on the index of the input coordinates in the rule table corresponding to each filter core, the activation input data is extracted from the input data and aggregated to obtain the activation input matrix corresponding to each filter core; the activation input matrix corresponding to each filter core is multiplied by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; based on the rule table corresponding to each filter core, the activation output data corresponding to each filter core is scattered to the corresponding output coordinates, and the output data under the same output coordinates are superimposed to obtain the operation result of the submanifold sparse convolution, that is, the output tensor.

[0109] In this embodiment, the offset is generated by matrix multiplication, which has low computational complexity and requires fewer memory access operations, is more in line with the characteristics of TPU, and improves the efficiency of offset calculation; combined with the symmetric activation property of the submanifold sparse convolution, after calculating the rule table of the previous filter kernel that is symmetrically activated, the rule table of the next filter kernel that is symmetrically activated is obtained by exchanging the index of the input coordinates with the index of the output coordinates in the rule table of the filter kernel, which greatly simplifies the steps of generating the rule table and improves the efficiency of generating the rule table.

[0110] In the method for generating a sub-stream sparse convolution rule table provided in an embodiment of the present application, since the performance of TPU global memory access is affected by the length of a single memory access data, in order to avoid a large number of memory access operations that result in reduced efficiency of sub-stream sparse convolution operations implemented on the TPU, a rule table is generated by replacing memory access-based mask screening with matrix multiplication-based mask screening, thereby greatly improving the efficiency of rule table generation.

[0111] The steps provided in the various method embodiments of the present application can be executed in parallel or in series if logic permits. The present application does not limit the order in which the steps are executed, and the order shown in the drawings is only an example.

[0112] An embodiment of the present application provides a sub-flow type sparse convolution operation method, including: based on the index of the input coordinates in the rule table corresponding to each filter core, extracting and aggregating the activation input data from the input data to obtain the activation input matrix corresponding to each filter core; multiplying the activation input matrix corresponding to each filter core with the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; based on the rule table corresponding to each filter core, scattering the activation output data corresponding to each filter core to the corresponding output coordinates, and superimposing the output data under the same output coordinates to obtain the operation result of the sub-flow type sparse convolution.

[0113] Among them, the rule table is obtained based on the method for generating the sub-flow type sparse convolution rule table provided in any embodiment of the present application.

[0114] Figure 7 A schematic diagram of a substream type sparse convolution operation process provided by an embodiment of the present application is shown in FIG. Figure 7 As shown, the sub-flow type sparse convolution operation process includes two stages, the rule table generation stage and the convolution operation stage.

[0115] In the rule table generation stage, first input the input coordinates of valid input data, where valid input data is a non-zero value in the input data; then generate the rule table, that is, generate the mapping relationship between the index of the input coordinates and the index of the output coordinates under each filter kernel, and the rule table can be generated based on the method provided by any of the aforementioned embodiments. In the convolution operation stage, input valid input data is input, and the input data that needs to participate in the calculation is aggregated through the index of the input coordinates in the rule table, that is, the input data is activated; the aggregated input data is multiplied by the weight matrix of the filter to obtain the output data; based on the index of the output coordinates in the rule table, the output data is scattered and superimposed to the specified position, that is, the position of the corresponding output coordinate.

[0116] Figure 8 A schematic diagram of a device for generating a substream type sparse convolution rule table provided in an embodiment of the present application, wherein the device for generating a substream type sparse convolution rule table is applied to a tensor processing unit, such as Figure 8 As shown, the device for generating the sub-flow type sparse convolution rule table includes: a first offset calculation module 810, a second offset calculation module 820, an offset comparison module 830 and a rule table generation module 840.

[0117] The first offset calculation module 810 is used to calculate the offset of the input coordinates; the second offset calculation module 820 is used to calculate the output coordinates corresponding to the input coordinates in each filter kernel, and calculate the offset of the output coordinates; the offset comparison module 830 is used to compare the offset of the input coordinates and the offset of the output coordinates corresponding to the filter kernel, and obtain the sequence mask corresponding to the filter kernel, wherein the elements in the sequence mask that are not 0 indicate that the offset of the corresponding set of input coordinates is the same as the offset of the output coordinates; the rule table generation module 840 is used to determine the rule table corresponding to the filter kernel based on the sequence mask corresponding to the filter kernel, so as to perform sub-flow type sparse convolution operation based on the rule table.

[0118] Optionally, the rule table generating module 840 is specifically used for: Based on the number of offsets of the input coordinates, an index sequence is generated, where the index sequence is an arithmetic progression starting from 0 and with a step length of 1; taking the sequence mask corresponding to the filter kernel as input and the mask, based on the mask screening function, determining the index of the output coordinate corresponding to the filter kernel; taking the sequence mask corresponding to the filter kernel as the mask and the index sequence as input, determining the index of the input coordinate corresponding to the filter kernel; establishing a mapping relationship between the index of the output coordinate corresponding to the filter kernel and the index of the input coordinate, and obtaining a rule table corresponding to the filter kernel.

[0119] Optionally, the filter core of the sub-flow type sparse convolution includes multiple groups of symmetrically activated first filter cores and second filter cores, and an intermediate filter core; the rule table generation module 840 includes: The first type of rule table generating unit is used to determine, for each first filter core, the rule table corresponding to the first filter core based on the sequence mask and the mask screening function corresponding to the first filter core; the second type of rule table generating unit is used to generate, for each second filter core, the rule table corresponding to the second filter core based on the rule table corresponding to the first filter core symmetrically activated with the second filter core; and the third type of rule table generating unit is used to determine the rule table corresponding to the intermediate filter core based on the offset of the input coordinates.

[0120] Optionally, the second type rule table generating unit is specifically used for: Taking the index of the input coordinate in the rule table corresponding to the first filter kernel as the index of the output coordinate, and taking the index of the output coordinate in the rule table corresponding to the first filter kernel as the index of the input coordinate, a mapping relationship between the index of the output coordinate and the index of the input coordinate corresponding to the second filter kernel is established to obtain the rule table corresponding to the filter of the second filter kernel.

[0121] Optionally, the offset comparison module 830 is specifically used for: The offset of the input coordinates and the offset of the output coordinates corresponding to the filter kernel are compared, and a first mask matrix is ​​obtained based on the comparison result; the first mask matrix is ​​left-multiplied by a preset sequence to obtain a sequence mask corresponding to the filter kernel.

[0122] Optionally, the first offset calculation module 810 is specifically configured to: Based on the dimension of the input coordinates, an offset vector is constructed; and the transpose of the offset vector is multiplied on the left by the vector corresponding to the input coordinates to obtain the offset of the input coordinates.

[0123] Optionally, the second offset calculation module 820 is specifically configured to: Calculate the output coordinates corresponding to the input coordinates in each filter kernel; construct an offset vector based on the dimension of the output coordinates; and multiply the transpose of the offset vector by the vector corresponding to the output coordinates to obtain the offset of the output coordinates.

[0124] The device for generating a sub-stream type sparse convolution rule table provided in an embodiment of the present application can execute the method for generating a sub-stream type sparse convolution rule table provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.

[0125] The embodiment of the present application further provides a sub-flow type sparse convolution operation device, which is applied to a tensor processing unit, including: An aggregation module is used to extract and aggregate activation input data from input data based on the index of input coordinates in the rule table corresponding to each filter core, so as to obtain an activation input matrix corresponding to each filter core; an output calculation module is used to multiply the activation input matrix corresponding to each filter core by the weight matrix of the corresponding filter core, so as to obtain activation output data corresponding to each filter core; a scattering and superposition module is used to scatter the activation output data corresponding to each filter core to the corresponding output coordinates based on the rule table corresponding to each filter core, and superimpose the output data under the same output coordinates to obtain the operation result of sub-flow type sparse convolution; wherein, the rule table is generated based on the method for generating the sub-flow type sparse convolution rule table provided in any embodiment of the present application.

[0126] An embodiment of the present application also provides a tensor processing unit, which is used to perform sub-flow type sparse convolution operations, and stores a sparse convolution rule table thereon, wherein the rule table is generated based on the method for generating the sub-flow type sparse convolution rule table provided in any embodiment of the present application.

[0127] The tensor processing unit is specifically used for: Based on the index of the input coordinates in the rule table corresponding to each filter core, the activation input data is extracted from the input data and aggregated to obtain the activation input matrix corresponding to each filter core; the activation input matrix corresponding to each filter core is multiplied by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; based on the rule table corresponding to each filter core, the activation output data corresponding to each filter core is scattered to the corresponding output coordinates, and the output data under the same output coordinates are superimposed to obtain the operation result of the sub-flow type sparse convolution.

[0128] An embodiment of the present application provides a computer-readable storage medium on which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method provided by any embodiment of the present application is implemented.

[0129] Among them, the computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0130] An embodiment of the present application provides a computer program product, including a computer program, which implements the method provided by any embodiment of the present application when the computer program is executed by a tensor processing unit.

[0131] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0132] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for generating a sub-flow type sparse convolution rule table, characterized in that: The method is applied to a tensor processing unit, and the method comprises: Calculate the offset of the input coordinates; Calculating the output coordinates corresponding to the input coordinates in each filter kernel, and calculating the offset of the output coordinates; Compare the offset of the input coordinates and the offset of the output coordinates corresponding to the filter kernel to obtain a sequence mask corresponding to the filter kernel, wherein an element in the sequence mask that is not 0 indicates that the offset of a corresponding set of input coordinates is the same as the offset of the output coordinates; Based on the sequence mask corresponding to the filter kernel and the mask screening function, a rule table corresponding to the filter kernel is obtained, so as to perform a sub-flow type sparse convolution operation based on the rule table.

2. The method according to claim 1, characterized in that Based on the sequence mask corresponding to the filter kernel and the mask screening function, a rule table corresponding to the filter kernel is obtained, including: Generate an index sequence based on the number of offsets of the input coordinates, where the index sequence is an arithmetic progression starting from 0 and with a step length of 1; Taking the sequence mask corresponding to the filter kernel as input and as the mask, based on the mask screening function, determining the index of the output coordinate corresponding to the filter kernel; Taking the sequence mask corresponding to the filter kernel as a mask and the index sequence as input, determining the index of the input coordinate corresponding to the filter kernel; A mapping relationship between the index of the output coordinate corresponding to the filter kernel and the index of the input coordinate is established to obtain a rule table corresponding to the filter kernel.

3. The method according to claim 1, characterized in that The filter kernel of the sub-flow type sparse convolution includes multiple groups of symmetrically activated first filter kernels and second filter kernels, and an intermediate filter kernel; Based on the sequence mask corresponding to the filter kernel and the mask screening function, a rule table corresponding to the filter kernel is obtained, including: For each first filter core, based on the sequence mask corresponding to the first filter core and the mask screening function, determining a rule table corresponding to the first filter core; For each second filter core, generating a rule table corresponding to the second filter core based on a rule table corresponding to a first filter core symmetrically activated with the second filter core; Based on the offset of the input coordinates, a rule table corresponding to the intermediate filter kernel is determined.

4. The method according to claim 3, characterized in that Generating a rule table corresponding to the second filter core based on a rule table corresponding to a first filter core that is symmetric to the second filter core includes: Taking the index of the input coordinate in the rule table corresponding to the first filter kernel as the index of the output coordinate, and taking the index of the output coordinate in the rule table corresponding to the first filter kernel as the index of the input coordinate, a mapping relationship between the index of the output coordinate and the index of the input coordinate corresponding to the second filter kernel is established to obtain the rule table corresponding to the filter of the second filter kernel.

5. The method according to claim 1, characterized in that Comparing the offset of the input coordinates with the offset of the output coordinates corresponding to the filter kernel to obtain a sequence mask corresponding to the filter kernel, including: Comparing the offset of the input coordinates with the offset of the output coordinates corresponding to the filter kernel, and obtaining a first mask matrix based on the comparison result; The first mask matrix is ​​left-multiplied by a preset sequence to obtain a sequence mask corresponding to the filter kernel.

6. The method according to any one of claims 1 to 5, characterized in that: Calculate the offset of input coordinates or output coordinates, including: constructing an offset vector based on the dimension of the input coordinate or the output coordinate; The transpose of the offset vector is multiplied on the left by the vector corresponding to the input coordinate or the vector corresponding to the output coordinate to obtain the offset of the input coordinate or the output coordinate.

7. A substream sparse convolution operation method, characterized in that: The method is applied to a tensor processing unit, comprising: Based on the index of the input coordinates in the rule table corresponding to each filter core, the activation input data is extracted from the input data and aggregated to obtain the activation input matrix corresponding to each filter core; Multiply the activation input matrix corresponding to each filter core by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; Based on the rule table corresponding to each filter core, the corresponding activation output data of each filter core is scattered to the corresponding output coordinate, and the output data under the same output coordinate is superimposed to obtain the operation result of sub-flow type sparse convolution; Wherein, the rule table is generated based on the method provided in any one of claims 1-6.

8. A device for generating a substream type sparse convolution rule table, characterized in that: The device is applied to a tensor processing unit, and the device comprises: A first offset calculation module, used to calculate the offset of the input coordinates; A second offset calculation module, used to calculate the output coordinates corresponding to the input coordinates in each filter kernel, and calculate the offset of the output coordinates; An offset comparison module, used for comparing the offset of the input coordinates and the offset of the output coordinates corresponding to the filter kernel, to obtain a sequence mask corresponding to the filter kernel, wherein the elements in the sequence mask that are not 0 indicate that the offset of the corresponding set of input coordinates is the same as the offset of the output coordinates; A rule table generating module is used to determine a rule table corresponding to the filter kernel based on a sequence mask corresponding to the filter kernel, so as to perform a sub-flow type sparse convolution operation based on the rule table.

9. A tensor processing unit, characterized in that: Used for performing substream type sparse convolution operation, a rule table of sparse convolution is stored thereon, and the rule table is generated based on the method provided in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a tensor processing unit.

Images