Sparse convolution rule table generation method, sparse convolution rule table operation method and tensor processing unit

Abstract

The invention provides a sparse convolution rule table generation method, an operation method and a tensor processing unit, and provides a sparse convolution rule table generation method for realizing sparse convolution on the tensor processing unit, and the method comprises the steps: calculating the offset of an input coordinate; calculating corresponding output coordinates of the input coordinates in each filter kernel, and calculating the offset of the output coordinates; the offset of the input coordinates and the offset of the output coordinates corresponding to the filter core are compared, a sequence mask corresponding to the filter core is obtained, and elements, not equal to 0, in the sequence mask indicate that the offset of the corresponding set of input coordinates is the same as the offset of the corresponding set of output coordinates; and obtaining a rule table corresponding to the filter core based on the sequence mask corresponding to the filter core and a mask screening function, and performing sparse convolution operation based on the rule table. The number of times of data access is effectively reduced, and the operation efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method for generating a sparse convolution rule table, a calculation method, and a tensor processing unit. Background Art

[0002] Detection algorithms based on point cloud data, since point cloud data is a typical sparse data, in deep learning algorithms based on point cloud data, sparse convolution (Spconv) operators are often used for convolution operations. The sparse convolution operator establishes the mapping relationship between the activation input data (non-zero data) and the output data index through the generated rule table, which greatly reduces the amount of calculation and memory access.

[0003] Common sparse convolution operators are implemented based on GPU (Garphics Processing Unit), which establishes a hash table of input-output tensor numbers, and then establishes a mapping relationship between input numbers and output numbers in the hash table based on the rule table. TPU (Tensor Processing Unit) is cheaper than GPU, and TPU is more suitable for neural network reasoning. In some scenarios, users choose to perform sparse convolution operations based on TPU. Due to the differences in TPU and GPU architecture and performance, the solution for generating rule tables in GPU cannot be directly transplanted to TPU.

[0004] Therefore, there is an urgent need to provide a solution for implementing sparse convolution operators on TPU. Summary of the invention

[0005] The present application provides a method for generating a sparse convolution rule table, a calculation method and a tensor processing unit, and implements a sparse convolution operator on a TPU.

[0006] In a first aspect, the present application provides a method for generating a sparse convolution rule table, which is applied to a tensor processing unit, and the method comprises: Calculate the output coordinates corresponding to the input coordinates in each filter kernel; For each filter kernel, based on a preset determination condition, determining whether the output coordinates corresponding to the filter kernel are valid, and obtaining a first mask corresponding to the filter kernel based on the determination result; Calculating the offset of each of the output coordinates; For a first filter kernel, based on a first mask corresponding to the first filter kernel, determining a mapping relationship between an index of an input coordinate and an index of an activated output coordinate corresponding to the first filter kernel, and obtaining a rule table corresponding to the first filter kernel; For each subsequent filter kernel, based on the first mask corresponding to the filter kernel and the comparison result of the offset of the output coordinate corresponding to the filter kernel and the offset of the activated output coordinate corresponding to the previous filter kernel, a mapping relationship between the index of the input coordinate and the index of the activated output coordinate corresponding to the filter kernel is determined to obtain a rule table corresponding to the filter kernel; The activated output coordinates are output coordinates that are determined to be valid.

[0007] In a second aspect, the present application provides a sparse convolution operation method, applied to a tensor processing unit, the method comprising: Based on the index of the input coordinates in the rule table corresponding to each filter core, the activation input data is extracted from the input data and aggregated to obtain the activation input matrix corresponding to each filter core; Multiply the activation input matrix corresponding to each filter core by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; Based on the rule table corresponding to each filter core, the corresponding activation output data of each filter core is scattered to the corresponding output coordinate, and the output data under the same output coordinate is superimposed to obtain the operation result of sparse convolution; Among them, the rule table is generated based on the sparse convolution rule table generation method provided in the first aspect of the present application.

[0008] In a third aspect, the present application provides a device for generating a sparse convolution rule table, which is applied to a tensor processing unit, and the device includes: Output coordinate calculation module, used to calculate the output coordinates corresponding to the input coordinates in each filter kernel; A first mask acquisition module is used to determine, for each filter kernel, whether the output coordinates corresponding to the filter kernel are valid based on a preset determination condition, and obtain a first mask corresponding to the filter kernel based on the determination result; An offset calculation module, used to calculate the offset of each of the output coordinates; A first rule table generating module is used to determine, for a first filter kernel, a mapping relationship between an index of an input coordinate and an index of an activated output coordinate corresponding to the first filter kernel based on a first mask corresponding to the first filter kernel, so as to obtain a rule table corresponding to the first filter kernel; A second rule table generating module is used to determine, for each subsequent filter kernel, a mapping relationship between an index of an input coordinate and an index of an activated output coordinate corresponding to the filter kernel based on a first mask corresponding to the filter kernel and a comparison result of an offset of an output coordinate corresponding to the filter kernel and an offset of an activated output coordinate corresponding to a previous filter kernel, so as to obtain a rule table corresponding to the filter kernel; The activated output coordinates are output coordinates that are determined to be valid.

[0009] In a fourth aspect, the present application provides a tensor processing unit for performing sparse convolution operations, on which a sparse convolution rule table is stored, and the rule table is generated based on the sparse convolution rule table generation method provided in the first aspect of the present application.

[0010] In a fifth aspect, the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a tensor processing unit, they are used to implement the method provided in the first aspect or the second aspect of the present application.

[0011] In a sixth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a tensor processing unit, implements the method provided in the first aspect or the second aspect of the present application.

[0012] The generation method, operation method and tensor processing unit of the sparse convolution rule table provided in this application realize sparse convolution operation on TPU. In the neural network application scenario, since TPU has a systolic array structure, it has better computing performance than GPU and improves the efficiency of convolution operation. In order to implement sparse convolution operation on TPU, a rule table generation method is provided, and the specific steps are as follows: calculating the output coordinates of the input coordinates under each filter core and judging whether the calculated output coordinates are valid, and obtaining the first mask corresponding to each filter core based on the judgment result; then calculating the offset of each output coordinate for subsequent offset comparison; first generating a rule table corresponding to the first filter core, and establishing a mapping relationship between the index of the activated output coordinate (valid output coordinate) corresponding to the first filter core and the index of the input coordinate based on the first mask and the mask filtering function; then for subsequent filter cores, by comparing the offset of the output coordinate corresponding to the filter core with the offset of the activated output coordinate corresponding to each previous filter core, the activated output coordinate of the filter core is divided into the newly added activated output coordinate and the existing output coordinate, and then by comparing the mask generated by the comparison process, the first mask and the mask filtering function, the mapping relationship between the index of the activated output coordinate (valid output coordinate) corresponding to the filter core and the index of the input coordinate is established to obtain a rule table, thereby implementing sparse convolution operation based on the rule table. The mask and mask screening functions are used to implement sparse convolution operations on the TPU, which effectively reduces the number of data memory accesses. It is more suitable for implementing sparse convolution operations on the TPU and improves computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0014] Figure 1 A schematic diagram of a point cloud data processing flow provided in an embodiment of the present application; Figure 2 A schematic diagram of a flow chart of a method for generating a sparse convolution rule table provided in one embodiment of the present application; Figure 3 A schematic diagram of a filter offset vector provided by one embodiment of the present application; Figure 4 For this application Figure 2 A schematic diagram of a first mask generation process in the illustrated embodiment; Figure 5 A schematic diagram of a second mask and a third mask generation process provided for one embodiment of the present application; Figure 6 A flowchart of a method for generating a sparse convolution rule table provided in another embodiment of the present application; Figure 7 For this application Figure 6 A schematic diagram of the calculation process of the third mask and the fourth mask in the illustrated embodiment; Figure 8 A schematic diagram of a process for generating a sequence number mapping array provided by an embodiment of the present application; Fig. 9 A schematic diagram of a sparse convolution operation process provided by an embodiment of the present application; Fig.10 A schematic diagram of a device for generating a sparse convolution rule table provided in one embodiment of the present application.

[0015] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0016] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0017] First, the terms involved in this application are explained: Mask: A matrix used to recalculate the values ​​of each element of a matrix, such as a 0-1 mask. The values ​​of the elements in the 0-1 mask are 0 or 1. By multiplying the 0-1 mask with the corresponding matrix, the elements in the matrix can be selected.

[0018] Mask filtering function: A function of the tensor processing unit that takes an array and a mask as input and extracts the element values ​​corresponding to the non-zero positions in the mask according to the index of these indexes in the input array.

[0019] Data aggregation function: input an index sequence and an array, based on the index in the index sequence, take out the values ​​from the corresponding index of the array and aggregate them together.

[0020] Point cloud data is widely used in many fields, such as target detection, 3D target reconstruction, and autonomous driving. Point cloud data is a typical sparse data. When processing point cloud data based on convolutional neural networks, since point cloud data includes a lot of data with values ​​of 0, in order to reduce the number of channels during convolution calculation, only non-zero values ​​in point cloud data are convolved, and sparse convolution is proposed.

[0021] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0022] Figure 1 A schematic diagram of a point cloud data processing flow provided in an embodiment of the present application is shown in FIG. Figure 1 As shown in the figure, taking the 3D target detection application scenario as an example, the point cloud data processing flow includes: collecting point cloud data, such as collecting point cloud data through lidar, 3D camera, etc.; preprocessing the point cloud data, such as noise reduction, registration, segmentation, etc.; inputting the preprocessed point cloud data into the convolutional neural network model for feature extraction and analysis, identifying the 3D target in the point cloud data, and realizing 3D target detection.

[0023] Convolutional neural networks for point cloud data usually use sparse convolution operators for convolution operations. Currently, the sparse convolution operator is only available on the GPU.

[0024] Compared with GPU, TPU has better computing performance due to its systolic array structure, which can improve the efficiency of convolution operations. Therefore, it is necessary to provide a solution for implementing sparse convolution operators for point cloud data on TPU.

[0025] The operation process of sparse convolution is divided into two steps. The first step is to generate a rule table (Rule Book). The second step is to aggregate the input data required by each filter core based on the generated rule table, input the aggregated input data into the corresponding filter core, and obtain the output data. Based on the output index in the rule table, the output data is dispersed to the corresponding position and the output data of different filters at this position are superimposed to realize the sparse convolution operation.

[0026] In order to adapt to the memory access characteristics of TPU, the embodiment of the present application provides a method for generating a sparse convolution rule table based on a mask, first calculating the output coordinates of the input coordinates under each filter core and calculating the offset of each output coordinate, judging whether the output coordinates are valid, and obtaining the first mask. Based on the first mask and the mask filtering function, a rule table corresponding to the first filter core is generated, and the subsequent filter cores are traversed. Based on the first mask, the mask filtering function, and the comparison result of the offset of the activated output coordinates corresponding to the current filter core and the offset of the activated output coordinates corresponding to the previous filter core, a rule table corresponding to the subsequent filter core is generated. By generating a rule table in a mask-based matrix calculation manner, the memory access operations required for rule table generation are reduced, the efficiency of rule table generation is improved, and thus the computational efficiency of sparse convolution based on TPU is improved.

[0027] Figure 2 A flow chart of a method for generating a sparse convolution rule table provided in one embodiment of the present application is provided. The method for generating a sparse convolution rule table is applied to a tensor processing unit, such as Figure 2 As shown, the method for generating the sparse convolution rule table includes the following steps: Step S201, calculating the output coordinates corresponding to the input coordinates in each filter kernel.

[0028] The input coordinates are the coordinates of the input data, such as the coordinates of the input data in the input tensor, which can be three-dimensional coordinates. Specifically, they can be the coordinates of the input data with non-zero values. The output coordinates are the coordinates of the output data, specifically the coordinates of the output data in the output tensor, which can be three-dimensional coordinates.

[0029] The calculation formula of the output coordinates is: Output coordinates = (input coordinates + supplementary vector - filter offset vector) / stride. Among them, the stride, filter offset vector and supplementary vector are all fixed values. The stride is greater than 1 and can be 2. The value of the supplementary vector in each dimension (such as x, y and z axis) is a fixed value, such as 1. The filter offset vector of the filter kernel is Figure 3 shown.

[0030] Figure 3 A schematic diagram of a filter offset vector provided by an embodiment of the present application, such as Figure 3 As shown, Figure 3Taking a 3*3*3 filter as an example, the filter offset vector of the filter kernel is marked on each filter kernel, such as (0, 0, 0), (0, 1, 0), etc.

[0031] The constant vector corresponding to each filter kernel can be calculated in advance, where the constant vector = supplementary vector - filter offset vector, and then the sum of the input coordinates and the constant vectors corresponding to each filter kernel is calculated, and then divided by the stride or multiplied by the inverse of the stride to obtain the output coordinates corresponding to the input coordinates under each filter kernel.

[0032] The input coordinates are stored in multiple local memories of the TPU. The addition calculation units based on each local memory can parallelly calculate the sum of the input coordinates stored thereon and the constant vectors corresponding to each filter kernel, and the multiplication calculation units based on the local memory can calculate the product of the sum value and the inverse of the stride to obtain the corresponding output coordinates.

[0033] For example, taking the input coordinates as (0, 2, 1), the supplementary vector as (1, 1, 1) and the stride as 2, the filter offset vector of the filter kernel is (1, 2, 0), and the corresponding output coordinates are (0, 0.5, 0.5).

[0034] Due to the lack of INT32 (32-bit integer) computing units on the TPU, the output coordinates can be calculated based on the INT16 computing units, and the output coordinates are represented by 16 integers (INT16). Compared with INT32, INT16 has higher computing power, thereby improving the computing speed of the output coordinates.

[0035] Step S202 : for each filter kernel, based on a preset determination condition, determining whether the output coordinates corresponding to the filter kernel are valid, and obtaining a first mask corresponding to the filter kernel based on the determination result.

[0036] The preset determination condition is a pre-designed determination condition used to determine whether the calculated output coordinates are valid, such as whether they fall within the spatial range of the corresponding filter kernel.

[0037] For each output coordinate corresponding to each calculated filter kernel, determine whether the output coordinate meets the preset judgment condition. If it does, the output coordinate is valid and is an activated output coordinate. Based on the judgment result of each output coordinate corresponding to a filter kernel, obtain the first mask corresponding to the filter kernel.

[0038] The first mask may be a 0-1 mask, and the value of an element in the 0-1 mask is 0 or 1. The output coordinates corresponding to the elements having a value of 1 in the first mask are activated output coordinates.

[0039] In some embodiments, the preset determination condition includes multiple conditions, the determination result of each condition corresponds to a mask, and an AND operation is performed on the masks corresponding to each condition to obtain the first mask.

[0040] The preset judgment conditions include spatial range judgment conditions and integer divisibility judgment conditions. The spatial range judgment conditions are used to judge whether the output coordinates are within the spatial range corresponding to the filter kernel. The integer divisibility judgment conditions are used to judge whether the first intermediate coordinates of the output coordinates can divide the stride. The first intermediate coordinates are equal to the input coordinates plus a constant vector. The constant vector is the difference vector obtained by subtracting the offset vector of the filter kernel from the supplementary vector.

[0041] For each filter kernel, it is determined whether each output coordinate corresponding to the filter kernel meets the spatial range determination condition and the integer divisibility determination condition, and the first sub-mask and the second sub-mask are obtained. The first sub-mask and the second sub-mask are both 0-1 masks. The output coordinates at the position of the element with a value of 1 in the first sub-mask meet the spatial range determination condition, and the output coordinates at the position of the element with a value of 1 in the second sub-mask meet the integer divisibility determination condition. The first sub-mask and the second sub-mask corresponding to the filter kernel are ANDed to obtain the first mask corresponding to the filter kernel.

[0042] Figure 4 For this application Figure 2 The schematic diagram of the first mask generation process in the embodiment shown is as follows: Figure 4 As shown, for the output coordinates corresponding to each filter kernel calculated, condition 1 is used to determine whether the x coordinate, y coordinate and z coordinate in the output coordinates are located in the corresponding spatial range, such as the x coordinate is between [0,21], the y coordinate is between [0,800], and the z coordinate is between [0,704]. The two masks corresponding to the x coordinate are obtained by comparing the x coordinate with the corresponding upper and lower boundaries, recorded as x1 mask and x2 mask. The two masks corresponding to the y coordinate are obtained by comparing the y coordinate with the corresponding upper and lower boundaries, recorded as y1 mask and mask y2. The two masks corresponding to the z coordinate are obtained by comparing the z coordinate with the upper and lower boundaries, recorded as z1 mask and mask z2. The mask satisfies the corresponding conditions and takes a value of 1, and does not satisfy the conditions and takes a value of 0. Through the judgment of condition 1, 6 masks (x1 mask, x2 mask, y1 mask, y2 mask, z1 mask and z2 mask) are obtained. The 6 masks are ANDed to obtain the mask corresponding to condition 1, which is recorded as mask 1. Through condition 2, it is judged whether the output coordinates can be divided by the stride, and the mask corresponding to condition 2 is obtained (1 means it can be divided, and 0 means it cannot be divided), which is recorded as mask 2. The mask corresponding to condition 1 is ANDed with the mask corresponding to condition 2 to obtain the first mask.

[0043] Step S203, calculating the offset of each output coordinate.

[0044] Different output coordinates have different offsets, and the offset value can be continuous, such as an arithmetic progression starting from 0 with a tolerance of 1. The offset is used to represent the position offset relationship between different output coordinates.

[0045] Specifically, the offset of each output coordinate may be generated based on the number of output coordinates.

[0046] The output coordinates can be k-dimensional coordinates, where k can be 2, 3, 4 or other values. The output coordinates are [a1, a2, ,ak] as an example, ai is a natural number in the interval [0, ni], i is a positive integer from 1 to k, and ni is a positive integer, then the output coordinates [a1, a2, , ak] The corresponding offset can be: , j is a positive integer greater than 1 and less than k.

[0047] Taking the four-dimensional coordinates of [a, b, c, d] as an example, where a, b, c and d are natural numbers in the intervals [0, n1], [0, n2], [0, n3] and [0, n4] respectively, the offset corresponding to the output coordinates [a, b, c, d] can be: a×n2×n3×n4+b×n3×n4+c×n4+d.

[0048] For example, taking the output coordinates as three-dimensional coordinates and the dimension of the output coordinates as [41, 1600, 1408] as an example, [41, 1600, 1408] means that the output coordinates are divided into 41 parts in the z-axis direction, 1600 parts in the y-axis direction, and 1408 parts in the x-axis direction. Then, the offset of the output coordinate (0, 1, 2) is: 0×1600×1408+1×1408+2×1=1410.

[0049] Specifically, each output coordinate can be evenly stored in multiple local memories of the TPU, such as 64 local memories; the offset of the output coordinate can be calculated in parallel in the independent computing unit corresponding to each local memory. Specifically, the offset of the output coordinate can be obtained by calling the INT32 matrix multiplication instruction, multiplying the vector corresponding to the output coordinate by a preset vector on the left.

[0050] Step S204, for the first filter kernel, based on the first mask corresponding to the first filter kernel, determine the mapping relationship between the index of the input coordinate and the index of the activated output coordinate corresponding to the first filter kernel, and obtain the rule table corresponding to the first filter kernel.

[0051] Traverse each filter core of the convolutional neural network to generate a rule table corresponding to each filter core. For the first filter core traversed, take the first mask corresponding to the first filter core as the mask, take the output coordinates and offsets corresponding to the first filter core as the input, and based on the mask screening function of TPU, obtain the activated output coordinates and offsets corresponding to the first filter core. The activated output coordinates are the output coordinates that are judged to be valid. Assign an index to the activated output coordinates corresponding to the first filter core, and based on the correspondence between the activated output coordinates and the input coordinates, obtain the rule table corresponding to the first filter core, that is, the mapping relationship between the index of the input coordinates and the index of the activated output coordinates corresponding to the first filter core.

[0052] The index of the input coordinates can be a continuous sequence, recorded as the input index sequence, such as a sequence starting from 0 and with a step length of 1. The index of the assigned output coordinates is also a continuous sequence, such as a sequence starting from 0 and with a step length of 1.

[0053] The first mask corresponding to the first filter kernel can be used as a mask and the input index sequence can be used as input. Based on the mask screening function of TPU, the index of the input coordinate corresponding to the index of the activated output coordinate corresponding to the first filter kernel can be obtained, that is, the rule table corresponding to the first filter kernel.

[0054] The extracted activation output coordinates and their offsets can be stored in the TPU's global memory for subsequent access.

[0055] Step S205, for each subsequent filter kernel, based on the first mask corresponding to the filter kernel and the comparison result of the offset of the output coordinate corresponding to the filter kernel and the offset of the activated output coordinate corresponding to the previous filter kernel, determine the mapping relationship between the index of the input coordinate and the index of the activated output coordinate corresponding to the filter kernel, and obtain the rule table corresponding to the filter kernel.

[0056] For each filter core traversed after the first filter core, such as the second filter core, the third filter core, etc., it is necessary to combine the first mask and the comparison result of the output coordinates corresponding to the filter core with the previously determined activation output coordinates to generate a rule table corresponding to the filter core.

[0057] Since the offset of the output coordinate is globally unique, it is possible to determine whether the output coordinate corresponding to the current filter kernel is the previously determined activated output coordinate, that is, whether it is the activated output coordinate of one of the previous filter kernels, based on the offset of the output coordinate.

[0058] The offset of the activated output coordinates can be copied from the global memory to the local memory, and the offset of the output coordinates of the current filter kernel is compared with the offset of the activated output coordinates. The second mask and the third mask are obtained based on the comparison result. The second mask is used to describe the newly added activated output coordinates of the current filter kernel compared to the previous filter kernel, and the third mask is used to describe the activated output coordinates of the current filter kernel that are the same as the previous filter kernel.

[0059] For example, Figure 5 A schematic diagram of a second mask and a third mask generation process provided by an embodiment of the present application is shown in FIG. Figure 5 As shown in the figure, the offset of the previous activation output coordinates is [3 4 7 11 12 14 16], and the offset of the output coordinates of the current filter kernel is [3 5 7 10 12 14 16]. By comparing the elements one by one, we know that the 1st, 3rd, 5th, 6th and 7th elements are the same, and the rest of the elements are different. Therefore, the second mask is determined to be [0 1 0 1 0 0 0], and the third mask is determined to be [1 0 1 0 1 1 1]. The values ​​of the elements at the same position in the second mask and the third mask are different.

[0060] Assign an index to the newly activated output coordinates and store the newly activated output coordinates and their offsets in global memory for subsequent access.

[0061] Through the first mask, the second mask, the third mask and the mask screening function, the mapping relationship between the index of the activated output coordinate corresponding to the current filter kernel and the index of the input coordinate is determined, that is, the rule table corresponding to the current filter kernel is obtained.

[0062] Since the performance of TPU global memory access is affected by the length of a single memory access data, in order to avoid a large number of memory access operations that would reduce the efficiency of sparse convolution operations implemented on the TPU, the rule table is generated by replacing the memory access-based mask filtering with matrix multiplication-based mask filtering, which greatly improves the efficiency of rule table generation.

[0063] The method for generating a sparse convolution rule table provided in this embodiment implements sparse convolution operations on a TPU. In a neural network application scenario, since the TPU has a systolic array structure, it has better computing performance than a GPU and improves the efficiency of convolution operations. In order to implement sparse convolution operation on TPU, a rule table generation method is provided, and the specific steps are as follows: calculating the output coordinates of the input coordinates under each filter core and judging whether the calculated output coordinates are valid, and obtaining the first mask corresponding to each filter core based on the judgment result; then calculating the offset of each output coordinate for subsequent offset comparison; first generating a rule table corresponding to the first filter core, and establishing a mapping relationship between the index of the activated output coordinate (valid output coordinate) corresponding to the first filter core and the index of the input coordinate based on the first mask and the mask filtering function; then for subsequent filter cores, by comparing the offset of the output coordinate corresponding to the filter core with the offset of the activated output coordinate corresponding to each previous filter core, the activated output coordinate of the filter core is divided into the newly added activated output coordinate and the existing output coordinate, and then by comparing the mask generated by the comparison process, the first mask and the mask filtering function, the mapping relationship between the index of the activated output coordinate (valid output coordinate) corresponding to the filter core and the index of the input coordinate is established to obtain a rule table, thereby implementing sparse convolution operation based on the rule table. The mask and mask screening functions are used to implement sparse convolution operations on the TPU, which effectively reduces the number of data memory accesses. It is more suitable for implementing sparse convolution operations on the TPU and improves computing efficiency.

[0064] Optionally, judging whether the output coordinates corresponding to the filter kernel are valid based on a preset judgment condition includes: For each output coordinate corresponding to the filter kernel, determine whether the output coordinate is within the spatial range corresponding to the filter kernel, and determine whether the first intermediate coordinate of the output coordinate can divide the stride. The first intermediate coordinate of the output coordinate is equal to the input coordinate corresponding to the output coordinate plus the constant vector corresponding to the filter kernel. The constant vector corresponding to the filter kernel is the difference vector obtained by subtracting the offset vector of the filter kernel from the supplementary vector. If both judgment results are yes, the output coordinate is valid.

[0065] The stride is the interval between the positions of the filter kernel in the input data. A larger stride means less calculation and computing time.

[0066] Specifically, it can be determined whether the value of the output coordinate in each dimension (such as the x-axis, y-axis and z-axis) is between the upper limit and lower limit (usually 0) of the filter kernel in each dimension. If so, the output coordinate is within the spatial range corresponding to the filter kernel.

[0067] When making a spatial range judgment, in order to improve the accuracy of the judgment, multiple judgments may be made, such as 2 times, 3 times or other times.

[0068] The first intermediate coordinate is the coordinate obtained by adding the corresponding constant vector to the input coordinate. The first intermediate coordinate is divided by the stride to obtain the output coordinate. If the first intermediate coordinate cannot divide the stride (usually 2), it means that the obtained output coordinate is invalid, that is, it is not an activated output coordinate.

[0069] By setting the spatial range and integer divisibility conditions, the calculated output coordinates are effectively judged, which realizes the reliable screening of the output coordinates, reduces the amount of output coordinates data that needs to be processed later, and improves the efficiency of rule table generation.

[0070] Optionally, determining whether the output coordinates are within a spatial range corresponding to the filter kernel includes: For each dimension of the output coordinate, determine at least twice whether the coordinate value of the output coordinate in the dimension is greater than or equal to 0 and less than or equal to the upper boundary of the spatial range corresponding to the filter kernel in the dimension; if the results of at least two judgments in each dimension are yes, the output coordinate is located in the spatial range corresponding to the filter kernel.

[0071] The output coordinates are three-dimensional coordinates, corresponding to the three dimensions of the x-axis, y-axis, and z-axis. The lower limit of the coordinate value of each dimension in the spatial range corresponding to the filter kernel is 0.

[0072] For the coordinate values ​​of each dimension of the output coordinate, it is repeatedly determined whether the coordinate value is between [0, max], where max is the upper boundary of the coordinate value under this dimension. If it is in this interval, the element value of the corresponding mask at the corresponding position of the coordinate is 1, otherwise it is 0. In this way, each of the three dimensions corresponds to two masks composed of 1 and 0. The AND operation is performed on the six masks to obtain the first sub-mask. The output coordinate corresponding to the position of 0 in the first sub-mask is an invalid coordinate.

[0073] The calculation related to the first sub-mask can be performed based on the INT8 (8-bit integer) computing unit in the TPU to save computing power.

[0074] Since division is not friendly to hardware devices, when determining the conditions related to integer division, the reciprocal of the stride can be calculated first, such as a 32-bit floating-point number, and the division can be replaced by multiplying the first intermediate coordinate by the reciprocal of the stride.

[0075] Optionally, determining whether the first intermediate coordinate of the output coordinate can divide the stride includes: Calculate the floating-point number corresponding to the reciprocal of the stride; convert the first intermediate coordinate of the output coordinate into a floating-point number and multiply it by the floating-point number corresponding to the reciprocal of the stride, organize the multiplication result into an integer to obtain the second intermediate coordinate; multiply the second intermediate coordinate by the stride to obtain the third intermediate coordinate; determine whether the first intermediate coordinate is consistent with the third intermediate coordinate; if they are consistent, the first intermediate coordinate of the output coordinate can divide the stride.

[0076] The floating point number corresponding to the reciprocal of the stride may be a 32-bit floating point number, and accordingly, the first intermediate coordinate of the output coordinate is converted into a 32-bit floating point number.

[0077] The 32-bit floating point number converted from the first intermediate coordinate is multiplied by the inverse of the stride (32 floating point number), and the result of the multiplication is rounded to INT16 (16-bit integer) to obtain the second intermediate coordinate (INT16), and the second intermediate coordinate is multiplied by the stride to obtain the third intermediate coordinate. If the third intermediate coordinate is equal to the first intermediate coordinate, the element at the corresponding position of the output coordinate of the second sub-mask is 1, otherwise it is 0, and the output coordinates corresponding to the filter kernel are traversed to obtain the second sub-mask corresponding to the filter kernel. The output coordinates corresponding to the elements with a value of 0 in the second sub-mask are invalid output coordinates.

[0078] By performing an AND operation on a group of first sub-masks and second sub-masks corresponding to the same filter kernel, the first mask corresponding to the filter kernel is obtained.

[0079] Figure 6 A flow chart of a method for generating a sparse convolution rule table provided in another embodiment of the present application. Figure 2 Based on the embodiment shown, step S204 and step S205 are further refined, as follows: Figure 6 As shown, the above steps S204 and S205 may include the following steps: Step S401, generating an input index sequence based on the number of input coordinates.

[0080] The input index array includes the index of each of the input coordinates.

[0081] The index sequence includes multiple indexes, and the number of indexes is equal to the number of offsets of the input coordinates of all input data (including input data with values ​​of 0 and non-zero).

[0082] Exemplarily, taking 15470 input coordinates as an example, the input index sequence is an arithmetic progression starting from 0, with a step length of 1, and a total of 15470 numbers.

[0083] Step S402, adding an index to the activation output coordinates corresponding to the first filter kernel.

[0084] The first filter kernel is the first filter kernel visited when traversing the filter kernels of the convolutional neural network.

[0085] The activated output coordinates are used to determine the valid output coordinates. The index of the activated output coordinates starts from 0 and the step length is 1.

[0086] The first mask corresponding to the filter kernel can be used as a mask, and the output coordinates corresponding to the filter kernel can be used as input. The activation output coordinates corresponding to the filter kernel can be obtained through the mask filtering function, and an index can be added to each activation output coordinate. Different activation output coordinates have different indexes.

[0087] The index added to the next activated output coordinate can be obtained by adding 1 to the index of the previous activated output coordinate.

[0088] Taking the case where there are 10 activation output coordinates corresponding to the first filter kernel as an example, the indexes of the 10 activation output coordinates are values ​​from 0 to 9 respectively.

[0089] Step S403 , using the first mask corresponding to the first filter kernel as a mask and the input index sequence as input, based on a mask screening function, obtain the index of the input coordinate corresponding to the first filter kernel.

[0090] Step S404, establishing a mapping relationship between the index of the activated output coordinate corresponding to the first filter core and the index of the input coordinate corresponding to the first filter core, and obtaining a rule table corresponding to the first filter core.

[0091] Through the mask screening function based on the first mask as the mask, a mapping relationship between the index of the activated output coordinate corresponding to the first filter core and the index of the input coordinate is obtained, and a rule table corresponding to the first filter core is obtained.

[0092] Step S405, for each subsequent filter kernel, based on the comparison result of the offset of the output coordinate corresponding to the filter kernel and the offset of the activated output coordinate corresponding to the previous filter kernel and the first mask corresponding to the filter kernel, generate the second mask and the third mask corresponding to the filter kernel.

[0093] The previous filter kernel is one or more filter kernels traversed before the filter kernel. The second mask is used to describe the newly added activation output coordinates of the filter kernel compared to the previous filter kernel, and the third mask is used to describe the activation output coordinates of the filter kernel that are the same as the previous filter kernel.

[0094] The newly added activated output coordinates and their offsets may be stored in the global memory. During comparison, the offsets of the stored activated output coordinates are read from the global memory to the local memory to obtain the offsets of the activated output coordinates corresponding to the previous filter kernel. The offsets of the output coordinates corresponding to the filter kernel are compared with the offsets of the activated output coordinates corresponding to the previous filter kernel in the local memory to obtain a comparison result, and based on the comparison result, the second mask and the third mask corresponding to the filter kernel are generated.

[0095] Optionally, the first mask, the second mask and the third mask are all 0-1 masks, the output coordinates corresponding to the elements with a value of 1 in the first mask are activated output coordinates, the output coordinates corresponding to the elements with a value of 1 in the second mask are newly activated output coordinates; the output coordinates corresponding to the elements with a value of 1 in the third mask are the same as the activated output coordinates corresponding to the previous filter kernel.

[0096] Optionally, based on a comparison result of the offset of the output coordinate corresponding to the filter kernel and the offset of the activated output coordinate corresponding to the previous filter kernel and the first mask corresponding to the filter kernel, generating a second mask and a third mask corresponding to the filter kernel includes: Based on the comparison result of the output coordinate offset corresponding to the filter kernel and the activated output coordinate offset corresponding to the previous filter kernel, a first mask matrix is ​​obtained; the first mask matrix is ​​left-multiplied by a preset sequence to obtain a fifth mask, and the values ​​of the elements greater than 0 in the fifth mask are converted to 1 to obtain the third mask; the first mask corresponding to the filter kernel is ANDed with the third mask corresponding to the filter kernel to obtain a fourth mask corresponding to the filter kernel; and the second mask corresponding to the filter kernel is determined to be the first mask corresponding to the filter kernel minus the fourth mask corresponding to the filter kernel.

[0097] Exemplarily, the preset sequence may be an arithmetic sequence, such as an arithmetic sequence starting from 1 and having a common difference of 1.

[0098] Figure 7 For this application Figure 6 The schematic diagram of the calculation process of the third mask and the fourth mask in the embodiment shown is as follows: Figure 7As shown, taking the offset of the output coordinates corresponding to the previous filter kernels as [5, 6, 11, 14, 16] and the offset of the output coordinates corresponding to the current filter kernel as [5, 7, 11, 12, 16] as an example, the offset of the output coordinates corresponding to the previous filter kernel is taken as a column vector, and the offset of the output coordinates corresponding to the current filter kernel is taken as a row vector. By comparing element by element, a first mask matrix (0-1 mask matrix) is obtained, where 0-1 indicates that the element value is 0 or 1. Then, the first mask matrix is ​​multiplied by a preset sequence, such as [1, 2, 3, 4, 5], to obtain the fifth mask corresponding to the current filter kernel, namely [1, 0, 3, 0,5]. The output coordinates at the non-zero element position in the fifth mask are one of the existing activated output coordinates. The elements greater than 0 in the fifth mask, namely 1, 3, and 5, are converted to 1 to obtain a 0-1 mask, [1, 0, 1, 0, 1], i.e., the third mask.

[0099] Continue to see Figure 7 , the first mask corresponding to the current filter kernel is [1,1,0,0,1], and the fourth mask is the mask obtained by the AND operation of the third mask and the first mask, that is, [1,0,0,0,1].

[0100] Since the third mask is obtained by comparing the offset, the output coordinate at position 1 in the third mask is theoretically an existing activated output coordinate. In order to improve the accuracy of the second mask, the first mask and the third mask are firstly ANDed. Ideally, the result of the ANDed operation is the third mask. Then, the first mask minus the result of the ANDed operation is calculated to obtain the second mask, which is equivalent to deleting the output coordinates that are the same as the previous activated output coordinates from the activated output coordinates corresponding to the filter kernel. Thus, the output coordinate at position 1 in the second mask is the newly added activated output coordinate of the filter kernel. The fourth mask = the first mask & the third mask, the second mask = the first mask - (the first mask & the third mask), where "&" represents the ANDed operation.

[0101] The second mask and the third mask are obtained in the above manner, which are respectively used to characterize the newly added activation output coordinates and the existing activation output coordinates in the current filter kernel, with high accuracy, and provide reliable masks for subsequent mask screening functions, thereby improving the accuracy of rule table generation.

[0102] In some embodiments, the newly added activation output coordinates of the filter core and their offsets may also be stored in the global memory, so that the activation output coordinates stored in the global memory can be accessed when a subsequent filter core rule table is generated.

[0103] The newly added activation output coordinates of the filter core can be obtained by using the second mask corresponding to the filter core as the mask and the output coordinates corresponding to the filter core as the input through the TPU mask screening function.

[0104] Optionally, the method further includes: The activation output coordinates and offsets corresponding to the first filter kernel, as well as the newly activated output coordinates and offsets corresponding to the subsequent filter kernels, are stored in the global memory; the offsets of the activation output coordinates corresponding to the previous filter kernel are read from the global memory to the local memory for offset comparison.

[0105] Step S406, taking the fourth mask corresponding to the filter core as the mask and the output coordinates corresponding to the filter core as the input, based on the mask screening function, obtain the newly added activated output coordinates of the filter core and assign an index.

[0106] The fourth mask corresponding to the filter kernel is a mask obtained by performing an AND operation on the first mask and the third mask corresponding to the filter kernel.

[0107] Step S407, taking the fourth mask corresponding to the filter core as the mask and the input index sequence as the input, based on the mask screening function, obtain the mapping relationship between the index of the newly added activated output coordinate of the filter core and the index of the input coordinate.

[0108] Step S408, taking the second mask corresponding to the filter kernel as the mask, taking the output coordinates corresponding to the filter kernel and the input index series as input respectively, based on the mask screening function, obtains the mapping relationship between the index of the activated output coordinates of the filter kernel and the index of the input coordinates which are the same as the previous filter kernel.

[0109] The mapping relationship between the index of the newly added activated output coordinates of the filter core and the index of the input coordinates and the mapping relationship between the index of the activated output coordinates of the filter core and the index of the input coordinates that are the same as the previous filter core constitute the rule table corresponding to the filter core.

[0110] Traverse each filter core and obtain the rule table corresponding to each filter core through the above steps.

[0111] In this embodiment, a mask matrix is ​​obtained by comparing the offsets, and a third mask and a second mask are obtained by a series of operations on the mask matrix. A rule table is obtained based on the first to third masks and the mask screening function. This method does not require excessive data memory access and has simple calculation operations. It can better adapt to the computing characteristics and memory access characteristics of the TPU, thereby improving the efficiency of sparse convolution operations implemented on the TPU.

[0112] In the rule table obtained by the above method, the index of the output coordinates is not sorted according to the offset of the output coordinates, but is sorted according to the order of appearance of the newly added activated output coordinates. Therefore, after obtaining the rule table corresponding to each filter core, the activated output coordinates in the rule table can be re-assigned indices according to the order of the offset of the output coordinates from small to large, so that the index of the activated output coordinate with a large offset in the rule table is greater than the index of the activated output coordinate with a small offset.

[0113] In order to achieve the index sorting by offset, the offsets of the activated output coordinates corresponding to the filter core can be sorted first, and the offsets of the activated output coordinates corresponding to the filter core before and after the sorting can be compared to obtain a 0-1 mask matrix. Through the 0-1 mask matrix, a mask representing the order of the index of the activated output coordinates in the new rule table is obtained. Based on the data aggregation function of TPU, the index of the original output coordinate in the rule table is replaced with the sorted new index indicated in the mask. The indexes in the rule table can also be rearranged in order from small to large.

[0114] Optionally, the method further includes: For each filter kernel, the offsets of the activated output coordinates corresponding to the filter kernel are sorted in order from small to large. Based on the position change of the offsets of the output coordinates before and after the sorting, a second mask matrix corresponding to the filter kernel is generated. Based on the second mask matrix corresponding to the filter kernel, the index of the activated output coordinates in the rule table corresponding to the filter kernel is updated, and the rule table corresponding to the filter kernel is rearranged in order from small to large according to the index of the activated output coordinates.

[0115] The second mask matrix is ​​a 0-1 matrix, with the vector corresponding to the offset after sorting as the column vector and the vector corresponding to the offset before sorting as the row vector. The offsets before and after sorting are compared element by element. If the i-th offset in the column vector is the same as the j-th offset in the row vector, the value of the element in the i-th row and j-th column of the second mask matrix is ​​1, otherwise it is 0.

[0116] Specifically, a new serial number of the index of the activated output coordinates may be determined based on the second mask matrix, and based on the new serial number, the order of the index of the activated output coordinates in the rule table may be adjusted to achieve rearrangement of the rule table.

[0117] Optionally, based on the second mask matrix corresponding to the filter kernel, updating the index of the activated output coordinate in the rule table corresponding to the filter kernel includes: The second mask matrix corresponding to the filter core is multiplied by a preset continuous sequence on the left to obtain a sequence mapping array; based on the value of each element of the sequence mapping array, the index of the activated output coordinate in the rule table corresponding to the filter core is updated.

[0118] The preset continuous sequence may be an arithmetic sequence starting from 0, with a tolerance of 1 and a length equal to the number of activated output coordinates corresponding to the filter kernel.

[0119] After obtaining the second mask matrix corresponding to the filter kernel, a preset continuous sequence is constructed, and the second mask matrix is ​​left-multiplied by the preset continuous sequence to obtain a sequence mapping array, in which each element is used to indicate a new sequence number of the index of the activation output coordinate of the corresponding position. Then, the data aggregation function of the TPU is used to adjust the order of the indexes in the rule table according to the order indicated by the sequence mapping array to obtain a new rule table.

[0120] For example, Figure 8 A schematic diagram of a process of generating a sequence number mapping array provided by an embodiment of the present application is shown in FIG. Figure 8 As shown, the vector corresponding to the offset of the output coordinate corresponding to the filter kernel is [16, 6, 11, 14, 5]. The vector corresponding to the offset after sorting is [5, 6, 11, 14, 16]. Then, the vector corresponding to the offset before sorting is used as the column vector, and the vector corresponding to the offset after sorting is used as the row vector. By comparing element by element, the second mask matrix is ​​obtained. Then, the second mask matrix is ​​multiplied on the left by an arithmetic progression [1, 2, 3, 4, 5], that is, a preset continuous sequence, to obtain a sequence mapping array, that is, [5, 2, 3, 4, 1], which means that the new sequence number of the original first index in the rule table is 5, that is, the fifth index, and the new sequence number of the original fifth index is 1, that is, the first index. Based on the data aggregation function of TPU, the original index in the rule table is replaced with the sorted index according to the sequence mask. Taking the index of the activated output coordinate in the rule table as [0 1 2 3 4] as an example, the serial number mapping array is adjusted to [4 1 2 3 0] to sort the index of the output coordinate in the rule table according to the offset.

[0121] Furthermore, after obtaining the rule table corresponding to each filter core, it also includes: Based on the index of the input coordinates in the rule table corresponding to each filter core, the activation input data is extracted from the input data and aggregated to obtain the activation input matrix corresponding to each filter core; the activation input matrix corresponding to each filter core is multiplied by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; based on the rule table corresponding to each filter core, the activation output data corresponding to each filter core is scattered to the corresponding output coordinates, and the output data under the same output coordinates are superimposed to obtain the sparse convolution operation result.

[0122] After obtaining multiple activation output data of the filter core, based on the index of the output coordinates corresponding to each activation data in the rule table, the multiple activation output data are scattered or dispersed to the corresponding output coordinates. The output coordinates corresponding to the activation output data are the output coordinates corresponding to the input coordinates of the input data for calculating the activation output data. When the activation output data of different filter cores correspond to the index of the same output coordinate, the activation output data of different filter cores are superimposed to obtain the final activation output data at the output coordinate. After the operation is completed, the operation result of the sparse convolution, i.e., the output tensor, is obtained.

[0123] Furthermore, the output tensor can also be sent to a downstream device or module for further analysis, such as target recognition based on the output tensor.

[0124] After obtaining the rule table, the sparse convolution operation result is obtained by aggregating the input data, scattering the output data, and superimposing it. The sparse convolution operation is implemented on the TPU, which improves the efficiency of the sparse convolution operation and reduces the operation cost.

[0125] The steps provided in the various method embodiments of the present application can be executed in parallel or in series if logic permits. The present application does not limit the order in which the steps are executed, and the order shown in the drawings is only an example.

[0126] The present application provides a sparse convolution operation method, including: Based on the index of the input coordinates in the rule table corresponding to each filter core, the activation input data is extracted from the input data and aggregated to obtain the activation input matrix corresponding to each filter core; the activation input matrix corresponding to each filter core is multiplied by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; based on the rule table corresponding to each filter core, the activation output data corresponding to each filter core is scattered to the corresponding output coordinates, and the output data under the same output coordinates are superimposed to obtain the sparse convolution operation result. Among them, the rule table is obtained based on the method for generating the sparse convolution rule table provided in any embodiment of the present application.

[0127] Fig. 9 A schematic diagram of a sparse convolution operation process provided by an embodiment of the present application is shown in FIG. Fig. 9 As shown, the sparse convolution operation process includes two stages, the rule table generation stage and the convolution operation stage.

[0128] In the rule table generation stage, for each filter core, the coordinates of the output data corresponding to the filter core (i.e., the output coordinates) are first calculated based on the coordinates of the input data (i.e., the input coordinates). Specifically, the coordinates of the output data can be calculated based on the addition calculation unit (Add) and the subtraction calculation unit (Sub); then, the CMP (Chip multiprocessors) instruction is used to check the legality of each calculated output coordinate to determine whether each output coordinate is valid, and the activated output coordinates or valid output coordinates corresponding to the filter core are obtained; the offset of the activated output coordinate corresponding to the filter core is calculated, and the offset of the confirmed activated output coordinates is updated based on the offset of the activated output coordinates corresponding to the filter core of each round of traversal. The confirmed activated output coordinates include the newly added activated output coordinates corresponding to each filter core; the offset of the activated output coordinate corresponding to the currently traversed filter core and the offset of the confirmed activated output coordinate are compared to obtain a 0-1 mask, and based on the 0-1 mask, the rule table corresponding to the filter core is generated by mask calculation. The specific process can be referred to the aforementioned embodiment.

[0129] In the convolution operation stage, valid input data is input, and the input data that needs to participate in the calculation is aggregated through the index of the input coordinates in the rule table, that is, the input data is activated; the aggregated input data is multiplied by the weight matrix of the filter kernel to obtain the output data; based on the index of the output coordinates in the rule table, the output data is scattered and superimposed to the specified position, that is, the position of the corresponding output coordinate.

[0130] Fig.10 A schematic diagram of a device for generating a sparse convolution rule table provided in an embodiment of the present application, wherein the device for generating a sparse convolution rule table is applied to a tensor processing unit, such as Fig.10 As shown, the device for generating the sparse convolution rule table includes: an output coordinate calculation module 810, a first mask acquisition module 820, an offset calculation module 830, a first rule table generation module 840 and a second rule table generation module 850.

[0131] The output coordinate calculation module 810 is used to calculate the output coordinates corresponding to the input coordinates in each filter core; the first mask acquisition module 820 is used to determine whether the output coordinates corresponding to the filter core are valid for each filter core based on preset judgment conditions, and obtain the first mask corresponding to the filter core based on the judgment result; the offset calculation module 830 is used to calculate the offset of each output coordinate; the first rule table generation module 840 is used to determine the mapping relationship between the index of the input coordinate and the index of the activated output coordinate corresponding to the first filter core based on the first mask corresponding to the first filter core, and obtain the rule table corresponding to the first filter core; the second rule table generation module 850 is used to determine the mapping relationship between the index of the input coordinate and the index of the activated output coordinate corresponding to the filter core for each subsequent filter core based on the first mask corresponding to the filter core and the comparison result of the offset of the output coordinate corresponding to the filter core and the offset of the activated output coordinate corresponding to the previous filter core, and obtain the rule table corresponding to the filter core; wherein, the activated output coordinate is the output coordinate judged to be valid.

[0132] Optionally, the first rule table generating module 840 is specifically configured to: Based on the number of input coordinates, an input index sequence is generated, wherein the input index sequence includes the index of each input coordinate; an index is added to the activated output coordinate corresponding to the first filter kernel; based on the mask filtering function, the first mask corresponding to the first filter kernel is used as a mask and the input index sequence is used as input to obtain the index of the input coordinate corresponding to the first filter kernel; a mapping relationship between the index of the activated output coordinate corresponding to the first filter kernel and the index of the input coordinate corresponding to the first filter kernel is established to obtain a rule table corresponding to the first filter kernel.

[0133] Optionally, the second rule table generating module 850 includes: The second and third mask generating units are used to generate the second mask and the third mask corresponding to the filter kernel based on the comparison result of the offset of the output coordinates corresponding to the filter kernel and the offset of the activated output coordinates corresponding to the previous filter kernel and the first mask corresponding to the filter kernel, wherein the second mask is used to describe the activated output coordinates newly added by the filter kernel compared with the previous filter kernel, and the third mask is used to describe the activated output coordinates of the filter kernel that are the same as the previous filter kernel; a new index assigning unit is added, which is used to use the fourth mask corresponding to the filter kernel as a mask and the output coordinates corresponding to the filter kernel as input, and based on the mask screening function, obtain the activated output coordinates newly added by the filter kernel and assign an index , wherein the fourth mask corresponding to the filter kernel is a mask obtained by performing an AND operation on the first mask and the third mask corresponding to the filter kernel; a newly added index mapping unit is used to use the fourth mask corresponding to the filter kernel as a mask, and the input index series as input, based on the mask screening function, to obtain the mapping relationship between the index of the newly added activated output coordinates of the filter kernel and the index of the input coordinates; an existing coordinate index mapping unit is used to use the second mask corresponding to the filter kernel as a mask, and the output coordinates corresponding to the filter kernel and the input index series as input, respectively, based on the mask screening function, to obtain the mapping relationship between the index of the activated output coordinates of the filter kernel that is the same as the previous filter kernel and the index of the input coordinates.

[0134] Optionally, the first mask, the second mask and the third mask are all 0-1 masks, the output coordinates corresponding to the elements with a value of 1 in the first mask are activated output coordinates, the output coordinates corresponding to the elements with a value of 1 in the second mask are newly activated output coordinates; the output coordinates corresponding to the elements with a value of 1 in the third mask are the same as the activated output coordinates corresponding to the previous filter kernel.

[0135] Accordingly, the second and third mask generating units are specifically used for: Based on the comparison result of the output coordinate offset corresponding to the filter kernel and the activated output coordinate offset corresponding to the previous filter kernel, a first mask matrix is ​​obtained; the first mask matrix is ​​left-multiplied by a preset sequence to obtain a fifth mask, and the values ​​of the elements greater than 0 in the fifth mask are converted to 1 to obtain the third mask; the first mask corresponding to the filter kernel is ANDed with the third mask corresponding to the filter kernel to obtain a fourth mask corresponding to the filter kernel; and the second mask corresponding to the filter kernel is determined to be the first mask corresponding to the filter kernel minus the fourth mask corresponding to the filter kernel.

[0136] Optionally, the device further comprises: The global storage module is used to store the activation output coordinates and offsets corresponding to the first filter core, as well as the newly activated output coordinates and offsets corresponding to the subsequent filter cores, into the global memory; the memory access module is used to read the offsets of the activation output coordinates corresponding to the previous filter core from the global memory to the local memory for offset comparison.

[0137] Optionally, the device further comprises: An offset sorting module is used to sort the offsets of the activated output coordinates corresponding to each filter core in the order of offset from small to large for each filter core; a second mask matrix generating module is used to generate a second mask matrix corresponding to the filter core based on the position change of the offsets of the output coordinates before and after sorting; an index updating module is used to update the index of the activated output coordinates in the rule table corresponding to the filter core based on the second mask matrix corresponding to the filter core.

[0138] Optional, index update module, specifically used to: The second mask matrix corresponding to the filter core is multiplied by a preset continuous sequence on the left to obtain a sequence mapping array; based on the value of each element of the sequence mapping array, the index of the activated output coordinate in the rule table corresponding to the filter core is updated.

[0139] Optionally, the first mask acquisition module 820 includes: A first condition determination unit is used to determine, for each output coordinate corresponding to each filter, whether the output coordinate is within the spatial range corresponding to the filter kernel; a second condition determination unit is used to determine, for each output coordinate corresponding to each filter, whether the first intermediate coordinate of the output coordinate can divide the stride, the first intermediate coordinate of the output coordinate is equal to the input coordinate corresponding to the output coordinate plus the constant vector corresponding to the filter kernel, the constant vector corresponding to the filter kernel is the difference vector obtained by subtracting the offset vector of the filter kernel from the supplementary vector; a first mask generation unit is used to generate a first mask corresponding to each filter kernel based on the judgment results output by the first condition determination unit and the second condition determination unit.

[0140] If the judgment results output by the first condition determination unit and the second condition determination unit are both yes, the corresponding output coordinates are valid, and the value of the element at the corresponding position in the first mask is 1.

[0141] Optionally, the first condition determination unit is specifically configured to: For each output coordinate corresponding to each filter, for each dimension of the output coordinate, determine at least twice whether the coordinate value of the output coordinate in the dimension is greater than or equal to 0 and less than or equal to the upper boundary of the spatial range corresponding to the filter kernel in the dimension; if the results of at least two judgments in each dimension are yes, then the output coordinate is located within the spatial range corresponding to the filter kernel.

[0142] Optionally, the second condition determination unit is specifically configured to: Calculate the floating-point number corresponding to the reciprocal of the stride; for each output coordinate corresponding to each filter, convert the first intermediate coordinate of the output coordinate into a floating-point number and multiply it by the floating-point number corresponding to the reciprocal of the stride, organize the multiplication result into an integer to obtain a second intermediate coordinate; multiply the second intermediate coordinate by the stride to obtain a third intermediate coordinate; determine whether the first intermediate coordinate is consistent with the third intermediate coordinate; if they are consistent, the first intermediate coordinate of the output coordinate can divide the stride.

[0143] The device for generating a sparse convolution rule table provided in the embodiment of the present application can execute the method for generating a sparse convolution rule table provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.

[0144] The embodiment of the present application further provides a sparse convolution operation device, which is applied to a tensor processing unit, including: An aggregation module is used to extract and aggregate activation input data from input data based on the index of input coordinates in the rule table corresponding to each filter core, so as to obtain an activation input matrix corresponding to each filter core; an output calculation module is used to multiply the activation input matrix corresponding to each filter core by the weight matrix of the corresponding filter core, so as to obtain activation output data corresponding to each filter core; a scattering and superposition module is used to scatter the activation output data corresponding to each filter core to the corresponding output coordinates based on the rule table corresponding to each filter core, and superimpose the output data under the same output coordinates to obtain a sparse convolution operation result; wherein the rule table is generated based on the sparse convolution rule table generation method provided in any embodiment of the present application.

[0145] An embodiment of the present application also provides a tensor processing unit, which is used to perform sparse convolution operations and stores a sparse convolution rule table thereon, wherein the rule table is generated based on the sparse convolution rule table generation method provided in any embodiment of the present application.

[0146] The tensor processing unit is specifically used for: Based on the index of the input coordinates in the rule table corresponding to each filter core, the activation input data is extracted from the input data and aggregated to obtain the activation input matrix corresponding to each filter core; the activation input matrix corresponding to each filter core is multiplied by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; based on the rule table corresponding to each filter core, the activation output data corresponding to each filter core is scattered to the corresponding output coordinates, and the output data under the same output coordinates are superimposed to obtain the sparse convolution operation result.

[0147] An embodiment of the present application provides a computer-readable storage medium on which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method provided by any embodiment of the present application is implemented.

[0148] Among them, the computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0149] An embodiment of the present application provides a computer program product, including a computer program, which implements the method provided by any embodiment of the present application when the computer program is executed by a tensor processing unit.

[0150] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0151] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for generating a sparse convolution rule table, characterized in that: The method is applied to a tensor processing unit, and the method comprises: Calculate the output coordinates corresponding to the input coordinates in each filter kernel; For each filter kernel, based on a preset determination condition, determining whether the output coordinates corresponding to the filter kernel are valid, and obtaining a first mask corresponding to the filter kernel based on the determination result; Calculating the offset of each of the output coordinates; For a first filter kernel, based on a first mask corresponding to the first filter kernel, determining a mapping relationship between an index of an input coordinate and an index of an activated output coordinate corresponding to the first filter kernel, and obtaining a rule table corresponding to the first filter kernel; For each subsequent filter kernel, based on the first mask corresponding to the filter kernel and the comparison result of the offset of the output coordinate corresponding to the filter kernel and the offset of the activated output coordinate corresponding to the previous filter kernel, a mapping relationship between the index of the input coordinate and the index of the activated output coordinate corresponding to the filter kernel is determined to obtain a rule table corresponding to the filter kernel; The activated output coordinates are output coordinates that are determined to be valid.

2. The method according to claim 1, characterized in that Based on the first mask corresponding to the first filter kernel, determining a mapping relationship between an index of an input coordinate and an index of an activated output coordinate corresponding to the first filter kernel, and obtaining a rule table corresponding to the first filter kernel, including: Based on the number of input coordinates, generate an input index sequence, the input index sequence including the index of each of the input coordinates; Adding indices to the activation output coordinates corresponding to the first filter kernel; Taking the first mask corresponding to the first filter kernel as the mask and the input index sequence as the input, based on the mask screening function, obtaining the index of the input coordinate corresponding to the first filter kernel; A mapping relationship between the index of the activated output coordinate corresponding to the first filter kernel and the index of the input coordinate corresponding to the first filter kernel is established to obtain a rule table corresponding to the first filter kernel.

3. The method according to claim 2, characterized in that Determining a mapping relationship between an index of an input coordinate and an index of an activated output coordinate corresponding to the filter kernel based on a comparison result of a first mask corresponding to the filter kernel and an offset of an output coordinate corresponding to the filter kernel and an offset of an activated output coordinate corresponding to a previous filter kernel, including: Generate a second mask and a third mask corresponding to the filter kernel based on a comparison result of the offset of the output coordinates corresponding to the filter kernel and the offset of the activated output coordinates corresponding to the previous filter kernel and a first mask corresponding to the filter kernel, wherein the second mask is used to describe the activated output coordinates newly added by the filter kernel compared to the previous filter kernel, and the third mask is used to describe the activated output coordinates of the filter kernel that are the same as those of the previous filter kernel; Taking the fourth mask corresponding to the filter kernel as the mask and the output coordinates corresponding to the filter kernel as the input, based on the mask screening function, obtaining the newly added activated output coordinates of the filter kernel and assigning an index, wherein the fourth mask corresponding to the filter kernel is a mask obtained by performing an AND operation on the first mask and the third mask corresponding to the filter kernel; Taking the fourth mask corresponding to the filter core as the mask and the input index sequence as the input, based on the mask screening function, obtaining the mapping relationship between the index of the newly added activated output coordinate of the filter core and the index of the input coordinate; Taking the second mask corresponding to the filter kernel as the mask, and taking the output coordinates corresponding to the filter kernel and the input index series as input respectively, based on the mask screening function, the mapping relationship between the index of the activated output coordinates of the filter kernel and the previous filter kernel and the index of the input coordinates is obtained.

4. The method according to claim 3, characterized in that The first mask, the second mask, and the third mask are all 0-1 masks, the output coordinates corresponding to the elements with a value of 1 in the first mask are activated output coordinates, the output coordinates corresponding to the elements with a value of 1 in the second mask are newly activated output coordinates; the output coordinates corresponding to the elements with a value of 1 in the third mask are the same as the activated output coordinates corresponding to the previous filter kernel; Based on the comparison result of the offset of the output coordinate corresponding to the filter kernel and the offset of the activated output coordinate corresponding to the previous filter kernel and the first mask corresponding to the filter kernel, a second mask and a third mask corresponding to the filter kernel are generated, including: Obtaining a first mask matrix based on a comparison result of the offset of the output coordinates corresponding to the filter kernel and the offset of the activated output coordinates corresponding to the previous filter kernel; Multiply the first mask matrix by a preset sequence to obtain a fifth mask, and convert the values ​​of elements greater than 0 in the fifth mask to 1 to obtain the third mask; Performing an AND operation on the first mask corresponding to the filter core and the third mask corresponding to the filter core to obtain a fourth mask corresponding to the filter core; The second mask corresponding to the filter core is determined to be the first mask corresponding to the filter core minus the fourth mask corresponding to the filter core.

5. The method according to claim 3, characterized in that: The method further comprises: Store the activation output coordinates and offsets corresponding to the first filter kernel, as well as the newly activated output coordinates and offsets corresponding to subsequent filter kernels, into global memory; Read the offset of the activation output coordinates corresponding to the previous filter kernel from the global memory to the local memory for offset comparison.

6. The method according to any one of claims 2 to 5, characterized in that: The method further comprises: For each filter core, sort the offsets of the activation output coordinates corresponding to the filter core in order from small to large offsets; Generate a second mask matrix corresponding to the filter kernel based on the position change of the offset of the output coordinates before and after the sorting; Based on the second mask matrix corresponding to the filter kernel, the index of the activated output coordinate in the rule table corresponding to the filter kernel is updated.

7. The method according to claim 6, characterized in that Based on the second mask matrix corresponding to the filter kernel, updating the index of the activated output coordinate in the rule table corresponding to the filter kernel comprises: Multiplying the filter kernel corresponding to the second mask matrix by a preset continuous number sequence to obtain a sequence mapping array; Based on the values ​​of the elements of the sequence number mapping array, the index of the activated output coordinate in the rule table corresponding to the filter core is updated.

8. The method according to any one of claims 1 to 5, characterized in that: Based on the preset determination condition, determining whether the output coordinates corresponding to the filter kernel are valid includes: For each output coordinate corresponding to the filter kernel, determine whether the output coordinate is within the spatial range corresponding to the filter kernel, and determine whether the first intermediate coordinate of the output coordinate can divide the stride, the first intermediate coordinate of the output coordinate is equal to the input coordinate corresponding to the output coordinate plus the constant vector corresponding to the filter kernel, and the constant vector corresponding to the filter kernel is a difference vector obtained by subtracting the offset vector of the filter kernel from the supplementary vector; If both judgment results are yes, the output coordinates are valid.

9. The method according to claim 8, characterized in that Determining whether the output coordinates are within the spatial range corresponding to the filter kernel includes: For each dimension of the output coordinate, determine at least twice whether the coordinate value of the output coordinate in the dimension is greater than or equal to 0 and less than or equal to the upper boundary of the spatial range corresponding to the filter kernel in the dimension; If the results of at least two judgments in each dimension are both yes, then the output coordinates are within the spatial range corresponding to the filter kernel.

10. The method according to claim 8, characterized in that Determining whether the first intermediate coordinate of the output coordinate can divide the stride, including: Calculate the floating point number corresponding to the reciprocal of the stride; Convert the first intermediate coordinate of the output coordinate into a floating point number and multiply it by the floating point number corresponding to the reciprocal of the stride, and convert the multiplication result into an integer to obtain a second intermediate coordinate; Multiply the second intermediate coordinate by the stride to obtain a third intermediate coordinate; Determining whether the first intermediate coordinate is consistent with the third intermediate coordinate; If they are consistent, the first intermediate coordinate of the output coordinate can divide the stride.

11. A sparse convolution operation method, characterized in that: The method is applied to a tensor processing unit, and the method comprises: Based on the index of the input coordinates in the rule table corresponding to each filter core, the activation input data is extracted from the input data and aggregated to obtain the activation input matrix corresponding to each filter core; Multiply the activation input matrix corresponding to each filter core by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; Based on the rule table corresponding to each filter core, the corresponding activation output data of each filter core is scattered to the corresponding output coordinate, and the output data under the same output coordinate is superimposed to obtain the operation result of sparse convolution; Wherein, the rule table is generated based on the method provided in any one of claims 1-10.

12. A tensor processing unit for performing sparse convolution operations, characterized in that: A rule table of sparse convolution is stored thereon, and the rule table is generated based on the method provided in any one of claims 1 to 10.

Images