Sparse convolution rule table generation method, calculation method and tensor processing unit

Abstract

The present application provides a method for generating a sparse convolution rule table, an operation method, and a tensor processing unit. To implement sparse convolution on a tensor processing unit, a method for generating a sparse convolution rule table is provided, comprising: calculating the offset of the input coordinates; calculating the output coordinates corresponding to the input coordinates in each filter kernel, and calculating the offset of the output coordinates; comparing the offset of the input coordinates with the offset of the output coordinates corresponding to the filter kernel to obtain a sequence mask corresponding to the filter kernel, wherein non-zero elements in the sequence mask indicate that the offset of a corresponding set of input coordinates and the offset of the output coordinates are the same; based on the sequence mask corresponding to the filter kernel and the mask filtering function, a rule table corresponding to the filter kernel is obtained, so as to perform a sparse convolution operation based on the rule table. This effectively reduces the number of data accesses and improves computational efficiency.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method for generating a sparse convolution rule table, an operation method, and a tensor processing unit. Background Art

[0002] Detection algorithms based on point cloud data are typically sparse data. Deep learning algorithms based on point cloud data often use the sparse convolution (Spconv) operator for convolution operations. The sparse convolution operator uses a generated rule table to establish a mapping relationship between the indexes of activated input data (non-zero data) and output data, significantly reducing computational complexity and memory access.

[0003] Common sparse convolution operators are implemented on GPUs (Graphics Processing Units). This involves creating a hash table of input and output tensor numbers, and then mapping the input and output numbers in the hash table based on a rule table. Tensor Processing Units (TPUs) are less expensive than GPUs and are more suitable for neural network inference. In some scenarios, users choose to perform sparse convolution operations on TPUs. However, due to differences in TPU architecture and performance, it's not possible to directly port the rule table generation method used on GPUs to TPUs.

[0004] Therefore, there is an urgent need to provide a solution for implementing sparse convolution operators on TPU. Summary of the Invention

[0005] This application provides a method for generating a sparse convolution rule table, an operation method, and a tensor processing unit, which implements a sparse convolution operator on a TPU.

[0006] In a first aspect, the present application provides a method for generating a sparse convolution rule table, which is applied to a tensor processing unit, and the method includes:

[0007] Calculate the output coordinates corresponding to the input coordinates in each filter kernel;

[0008] For each filter kernel, based on a preset determination condition, determining whether the output coordinates corresponding to the filter kernel are valid, and obtaining a first mask corresponding to the filter kernel based on the determination result;

[0009] Calculating the offset of each of the output coordinates;

[0010] For a first filter kernel, determining, based on a first mask corresponding to the first filter kernel, a mapping relationship between an index of an input coordinate and an index of an activated output coordinate corresponding to the first filter kernel, and obtaining a rule table corresponding to the first filter kernel;

[0011] For each subsequent filter kernel, determining a mapping relationship between an index of the input coordinate and an index of the activated output coordinate corresponding to the filter kernel based on the first mask corresponding to the filter kernel and a comparison result of the offset of the output coordinate corresponding to the filter kernel and the offset of the activated output coordinate corresponding to the previous filter kernel, and obtaining a rule table corresponding to the filter kernel;

[0012] The activated output coordinates are output coordinates that are determined to be valid.

[0013] In a second aspect, the present application provides a sparse convolution operation method applied to a tensor processing unit, the method comprising:

[0014] Based on the index of the input coordinates in the rule table corresponding to each filter core, the activation input data is extracted from the input data and aggregated to obtain the activation input matrix corresponding to each filter core;

[0015] Multiply the activation input matrix corresponding to each filter core by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core;

[0016] Based on the rule table corresponding to each filter core, the activation output data corresponding to each filter core is scattered to the corresponding output coordinate, and the output data under the same output coordinate is superimposed to obtain the sparse convolution operation result;

[0017] Among them, the rule table is generated based on the sparse convolution rule table generation method provided in the first aspect of this application.

[0018] In a third aspect, the present application provides a device for generating a sparse convolution rule table, which is applied to a tensor processing unit, and the device includes:

[0019] Output coordinate calculation module, used to calculate the output coordinates corresponding to the input coordinates in each filter kernel;

[0020] a first mask acquisition module, configured to determine, for each filter kernel, whether the output coordinates corresponding to the filter kernel are valid based on a preset determination condition, and obtain a first mask corresponding to the filter kernel based on the determination result;

[0021] An offset calculation module, used to calculate the offset of each of the output coordinates;

[0022] A first rule table generating module is configured to determine, for a first filter kernel, based on a first mask corresponding to the first filter kernel, a mapping relationship between an index of an input coordinate and an index of an activated output coordinate corresponding to the first filter kernel, and obtain a rule table corresponding to the first filter kernel;

[0023] a second rule table generating module, configured to determine, for each subsequent filter kernel, a mapping relationship between an index of an input coordinate and an index of an activated output coordinate corresponding to the filter kernel based on a first mask corresponding to the filter kernel and a comparison result of an offset of an output coordinate corresponding to the filter kernel and an offset of an activated output coordinate corresponding to a previous filter kernel, thereby obtaining a rule table corresponding to the filter kernel;

[0024] The activated output coordinates are output coordinates that are determined to be valid.

[0025] In a fourth aspect, the present application provides a tensor processing unit for performing sparse convolution operations, on which a sparse convolution rule table is stored, and the rule table is generated based on the sparse convolution rule table generation method provided in the first aspect of the present application.

[0026] In a fifth aspect, the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a tensor processing unit, they are used to implement the method provided in the first aspect or the second aspect of the present application.

[0027] In a sixth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a tensor processing unit, implements the method provided in the first or second aspect of the present application.

[0028] This application provides a method for generating and computing a sparse convolution rule table, as well as a tensor processing unit (TPU). This method implements sparse convolution operations on the Tensor Processing Unit (TPU). In neural network applications, the TPU, with its systolic array structure, offers superior computing performance compared to GPUs, improving the efficiency of convolution operations. In order to implement sparse convolution operations on TPUs, a rule table generation method is provided, which specifically comprises the following steps: calculating the output coordinates of the input coordinates under each filter kernel and judging whether the calculated output coordinates are valid, and obtaining the first mask corresponding to each filter kernel based on the judgment result; then calculating the offset of each output coordinate for subsequent offset comparison; first generating a rule table corresponding to the first filter kernel, and establishing a mapping relationship between the index of the activated output coordinate (valid output coordinate) corresponding to the first filter kernel and the index of the input coordinate based on the first mask and the mask filtering function; then, for subsequent filter kernels, by comparing the offset of the output coordinate corresponding to the filter kernel with the offset of the activated output coordinate corresponding to each previous filter kernel, the activated output coordinate of the filter kernel is divided into newly activated output coordinates and existing output coordinates; and then, by using the mask generated in the comparison process, the first mask, and the mask filtering function, a mapping relationship between the index of the activated output coordinate (valid output coordinate) corresponding to the filter kernel and the index of the input coordinate is established to obtain a rule table, thereby implementing sparse convolution operations based on the rule table. The mask and mask filtering functions are used to implement sparse convolution operations on the TPU, effectively reducing the number of data memory accesses. This makes it more suitable for implementing sparse convolution operations on the TPU and improves computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0030] Figure 1 A schematic diagram of a point cloud data processing process provided in an embodiment of the present application;

[0031] Figure 2 A flowchart of a method for generating a sparse convolution rule table provided in one embodiment of the present application;

[0032] Figure 3 A schematic diagram of a filter offset vector provided in one embodiment of the present application;

[0033] Figure 4 For this application Figure 2 A schematic diagram of a first mask generation process in the illustrated embodiment;

[0034] Figure 5 A schematic diagram of a second mask and a third mask generation process provided in one embodiment of the present application;

[0035] Figure 6 A flowchart of a method for generating a sparse convolution rule table provided in another embodiment of the present application;

[0036] Figure 7 For this application Figure 6 A schematic diagram of the calculation process of the third mask and the fourth mask in the illustrated embodiment;

[0037] Figure 8 A schematic diagram of a process for generating a sequence number mapping array provided in one embodiment of the present application;

[0038] Figure 9 A schematic diagram of a sparse convolution operation process provided in one embodiment of the present application;

[0039] Figure 10 A schematic diagram of a device for generating a sparse convolution rule table provided in one embodiment of the present application.

[0040] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0041] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0042] First, let’s explain the terms involved in this application:

[0043] Mask: A matrix used to recalculate the values ​​of each element in a matrix. For example, a 0-1 mask has elements with values ​​of 0 or 1. By multiplying the 0-1 mask with the corresponding matrix, you can select elements in the matrix.

[0044] Mask filtering function: A function of the tensor processing unit that takes an array and a mask as input and extracts the element values ​​corresponding to the non-zero positions in the mask based on the index of the input array.

[0045] Data aggregation function: input an index sequence and an array, based on the index in the index sequence, extract the values ​​from the corresponding index of the array and aggregate them together.

[0046] Point cloud data is widely used in various fields, such as target detection, 3D object reconstruction, and autonomous driving. Point cloud data is a typical sparse data type. When processing point cloud data based on convolutional neural networks, since point cloud data often contains zero values, sparse convolution is proposed to reduce the number of channels required for convolution calculations by performing convolution only on non-zero values ​​in the point cloud data.

[0047] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0048] Figure 1 A schematic diagram of a point cloud data processing process provided in an embodiment of the present application is shown as follows: Figure 1 As shown in the figure, taking the 3D target detection application scenario as an example, the point cloud data processing flow includes: collecting point cloud data, such as collecting point cloud data through lidar, 3D camera, etc.; preprocessing the point cloud data, such as noise reduction, registration, segmentation, etc.; inputting the preprocessed point cloud data into the convolutional neural network model for feature extraction and analysis, identifying the 3D targets in the point cloud data, and realizing 3D target detection.

[0049] Convolutional neural networks for point cloud data usually use sparse convolution operators for convolution operations. Currently, the sparse convolution operator is only implemented on the GPU.

[0050] Compared with GPUs, TPUs have better computing performance due to their systolic array structure, which can improve the efficiency of convolution operations. Therefore, it is necessary to provide a solution for implementing sparse convolution operators for point cloud data on TPUs.

[0051] The operation process of sparse convolution is divided into two steps. The first step is to generate a rule table (Rule Book). The second step is to aggregate the input data required by each filter core based on the generated rule table, input the aggregated input data into the corresponding filter core, and obtain the output data. Based on the output index in the rule table, the output data is dispersed to the corresponding position and the output data of different filters at this position are superimposed to realize the sparse convolution operation.

[0052] In order to adapt to the memory access characteristics of the TPU, an embodiment of the present application provides a method for generating a mask-based sparse convolution rule table. First, the output coordinates of the input coordinates under each filter core are calculated and the offset of each output coordinate is calculated. Then, whether the output coordinates are valid is determined to obtain a first mask. Based on the first mask and the mask filtering function, a rule table corresponding to the first filter core is generated. Then, each subsequent filter core is traversed. Based on the first mask, the mask filtering function, and the comparison result of the offset of the activated output coordinates corresponding to the current filter core and the offset of the activated output coordinates corresponding to the previous filter core, a rule table corresponding to the subsequent filter core is generated. By generating a rule table through a mask-based matrix calculation method, the memory access operations required for rule table generation are reduced, the efficiency of rule table generation is improved, and thus the computational efficiency of sparse convolution implemented based on the TPU is improved.

[0053] Figure 2 A flow chart of a method for generating a sparse convolution rule table provided in one embodiment of the present application is provided. The method for generating a sparse convolution rule table is applied to a tensor processing unit, such as Figure 2 As shown, the method for generating the sparse convolution rule table includes the following steps:

[0054] Step S201 : Calculate the output coordinates corresponding to the input coordinates in each filter kernel.

[0055] Input coordinates are the coordinates of the input data, such as the coordinates of the input data within the input tensor, and can be 3D coordinates. Specifically, they can be non-zero coordinates of the input data. Output coordinates are the coordinates of the output data, specifically the coordinates of the output data within the output tensor, and can be 3D coordinates.

[0056] The formula for calculating the output coordinates is: Output coordinates = (Input coordinates + Supplement vector - Filter offset vector) / Stride. Among them, the stride, filter offset vector and supplement vector are all fixed values. The stride is greater than 1 and can be 2. The value of the supplement vector in each dimension (such as x, y and z axis) is a fixed value, such as 1. The filter offset vector of the filter kernel is as follows Figure 3 shown.

[0057] Figure 3 A schematic diagram of a filter offset vector provided in one embodiment of the present application is shown in FIG. Figure 3 As shown, Figure 3 Taking a 3*3*3 filter as an example, the filter offset vector of the filter kernel is marked on each filter kernel, such as (0, 0, 0), (0, 1, 0), etc.

[0058] The constant vector corresponding to each filter kernel can be calculated in advance, where the constant vector = the supplementary vector - the filter offset vector. Then, the sum of the input coordinates and the constant vectors corresponding to each filter kernel is calculated, and then divided by the stride or multiplied by the inverse of the stride to obtain the output coordinates corresponding to the input coordinates under each filter kernel.

[0059] The input coordinates are stored in multiple local memories of the TPU. The addition calculation unit based on each local memory can parallelly calculate the sum of the input coordinates stored thereon and the constant vector corresponding to each filter kernel, and the multiplication calculation unit based on the local memory can calculate the product of the sum value and the inverse of the stride to obtain the corresponding output coordinates.

[0060] For example, taking the input coordinates as (0, 2, 1), the supplementary vector as (1, 1, 1) with a stride of 2, the filter offset vector of the filter kernel is (1, 2, 0), and the corresponding output coordinates are (0, 0.5, 0.5).

[0061] Due to the lack of INT32 (32-bit integer) computing units on the TPU, the output coordinates can be calculated based on the INT16 computing unit, and the output coordinates are represented by 16 integers (INT16). Compared with INT32, INT16 has higher computing power, thereby improving the calculation speed of the output coordinates.

[0062] In step S202 , for each filter kernel, based on a preset determination condition, it is determined whether the output coordinates corresponding to the filter kernel are valid, and a first mask corresponding to the filter kernel is obtained based on the determination result.

[0063] The preset determination condition is a pre-designed determination condition used to determine whether the calculated output coordinates are valid, such as whether they fall within the spatial range of the corresponding filter kernel.

[0064] For each calculated output coordinate corresponding to each filter kernel, determine whether the output coordinate meets the preset judgment condition. If so, the output coordinate is valid and becomes an activated output coordinate. Based on the judgment results of each output coordinate corresponding to a filter kernel, a first mask corresponding to the filter kernel is obtained.

[0065] The first mask may be a 0-1 mask, where the value of an element in the 0-1 mask is 0 or 1. The output coordinates corresponding to the elements having a value of 1 in the first mask are activated output coordinates.

[0066] In some embodiments, the preset determination condition includes multiple conditions, the determination result of each condition corresponds to a mask, and an AND operation is performed on the masks corresponding to the conditions to obtain the first mask.

[0067] The preset judgment conditions include spatial range judgment conditions and divisibility judgment conditions. The spatial range judgment conditions are used to judge whether the output coordinates are within the spatial range corresponding to the filter kernel. The divisibility judgment conditions are used to judge whether the first intermediate coordinate of the output coordinates can divide the stride. The first intermediate coordinate is equal to the input coordinate plus a constant vector. The constant vector is the difference vector obtained by subtracting the offset vector of the filter kernel from the supplementary vector.

[0068] For each filter kernel, determine whether the output coordinates corresponding to the filter kernel meet the spatial range determination condition and the divisibility determination condition, respectively, to obtain the first sub-mask and the second sub-mask. The first sub-mask and the second sub-mask are both 0-1 masks. The output coordinates at the positions of the elements with a value of 1 in the first sub-mask meet the spatial range determination condition, and the output coordinates at the positions of the elements with a value of 1 in the second sub-mask meet the divisibility determination condition. An AND operation is performed on the first and second sub-masks corresponding to the filter kernel to obtain the first mask corresponding to the filter kernel.

[0069] Figure 4 For this application Figure 2 The schematic diagram of the first mask generation process in the embodiment shown is as follows: Figure 4 As shown, for the output coordinates corresponding to each filter kernel, condition 1 is used to determine whether the x, y, and z coordinates in the output coordinates are within the corresponding spatial range. For example, the x coordinate is between [0, 21], the y coordinate is between [0, 800], and the z coordinate is between [0, 704]. The two masks corresponding to the x coordinate are obtained by comparing the x coordinate with the corresponding upper and lower boundaries, and are recorded as x1 mask and x2 mask. The two masks corresponding to the y coordinate are obtained by comparing the y coordinate with the corresponding upper and lower boundaries, and are recorded as y1 mask and mask y2. The two masks corresponding to the z coordinate are obtained by comparing the z coordinate with the upper and lower boundaries, and are recorded as z1 mask and mask z2. The value of the mask that meets the corresponding condition is 1, and the value that does not meet the condition is 0. Based on condition 1, we obtain six masks (x1 mask, x2 mask, y1 mask, y2 mask, z1 mask, and z2 mask). We perform an AND operation on these six masks to obtain the mask corresponding to condition 1, denoted as Mask 1. Based on condition 2, we determine whether the output coordinates are divisible by the stride, obtaining the mask corresponding to condition 2 (1 indicates divisibility, 0 indicates non-divisibility), denoted as Mask 2. We perform an AND operation on the masks corresponding to condition 1 and condition 2 to obtain the first mask.

[0070] Step S203: Calculate the offset of each output coordinate.

[0071] Different output coordinates have different offsets. The offset value can be continuous, such as an arithmetic progression starting from 0 with a tolerance of 1. The offset is used to represent the positional offset relationship between different output coordinates.

[0072] Specifically, the offset of each output coordinate may be generated based on the number of output coordinates.

[0073] The output coordinates can be k-dimensional coordinates, where k can be 2, 3, 4 or other values. The output coordinates are [a1, a2, ,ak] as an example, ai is a natural number in the interval [0, ni], i is a positive integer from 1 to k, ni is a positive integer, then the output coordinates [a1, a2, , ak] The corresponding offset can be: , j is a positive integer greater than 1 and less than k.

[0074] Taking the four-dimensional output coordinates [a, b, c, d] as an example, where a, b, c, and d are natural numbers in the intervals [0, n1], [0, n2], [0, n3], and [0, n4] respectively, the offset corresponding to the output coordinates [a, b, c, d] can be: a×n2×n3×n4+b×n3×n4+c×n4+d.

[0075] For example, taking the output coordinates as three-dimensional coordinates and the dimension of the output coordinates as [41, 1600, 1408] as an example, [41, 1600, 1408] means that the output coordinates are divided into 41 parts in the z-axis direction, 1600 parts in the y-axis direction, and 1408 parts in the x-axis direction. Then the offset of the output coordinate (0, 1, 2) is: 0×1600×1408+1×1408+2×1=1410.

[0076] Specifically, each output coordinate can be evenly stored in multiple local memories of the TPU, such as 64 local memories; the offset of the output coordinate is calculated in parallel in the independent computing unit corresponding to each local memory. Specifically, the offset of the output coordinate can be obtained by calling the INT32 matrix multiplication instruction, multiplying the vector corresponding to the output coordinate by a preset vector on the left.

[0077] Step S204 : For the first filter kernel, based on the first mask corresponding to the first filter kernel, determine the mapping relationship between the index of the input coordinate and the index of the activated output coordinate corresponding to the first filter kernel, and obtain a rule table corresponding to the first filter kernel.

[0078] Traverse each filter core of the convolutional neural network to generate a rule table corresponding to each filter core. For the first filter core traversed, use the first mask corresponding to the first filter core as the mask, and the output coordinates and offset corresponding to the first filter core as input. Based on the mask screening function of the TPU, obtain the activated output coordinates and offset corresponding to the first filter core. The activated output coordinates are the output coordinates that are judged to be valid. Assign an index to the activated output coordinates corresponding to the first filter core, and based on the correspondence between the activated output coordinates and the input coordinates, obtain the rule table corresponding to the first filter core, that is, the mapping relationship between the index of the input coordinates and the index of the activated output coordinates corresponding to the first filter core.

[0079] The index of the input coordinates can be a continuous sequence, recorded as the input index sequence, such as a sequence starting from 0 and with a step size of 1. The index of the assigned output coordinates is also a continuous sequence, such as a sequence starting from 0 and with a step size of 1.

[0080] The first mask corresponding to the first filter kernel can be used as a mask, and the input index sequence can be used as input. Based on the mask screening function of the TPU, the index of the input coordinate corresponding to the index of the activated output coordinate corresponding to the first filter kernel can be obtained, that is, the rule table corresponding to the first filter kernel.

[0081] The extracted activation output coordinates and their offsets can be stored in the TPU's global memory for subsequent access.

[0082] Step S205: For each subsequent filter kernel, based on the first mask corresponding to the filter kernel and the comparison result of the offset of the output coordinate corresponding to the filter kernel and the offset of the activated output coordinate corresponding to the previous filter kernel, determine the mapping relationship between the index of the input coordinate and the index of the activated output coordinate corresponding to the filter kernel, and obtain the rule table corresponding to the filter kernel.

[0083] For each filter core traversed after the first filter core, such as the second filter core, the third filter core, etc., it is necessary to combine the first mask and the comparison result of the output coordinates corresponding to the filter core with the previously determined activation output coordinates to generate a rule table corresponding to the filter core.

[0084] Since the offset of the output coordinate is globally unique, it is possible to determine whether the output coordinate corresponding to the current filter kernel is the previously determined activated output coordinate based on the offset of the output coordinate, that is, whether it is the activated output coordinate of one of the previous filter kernels.

[0085] The offset of the activation output coordinate can be copied from the global memory to the local memory, and the offset of the output coordinate of the current filter kernel is compared with the offset of the activation output coordinate. The second mask and the third mask are obtained based on the comparison result. The second mask is used to describe the activation output coordinates newly added by the current filter kernel compared to the previous filter kernel, and the third mask is used to describe the activation output coordinates that are the same as the previous filter kernel.

[0086] For example, Figure 5 A schematic diagram of the second mask and third mask generation process provided in one embodiment of the present application is shown in FIG. Figure 5 As shown in the figure, the offset of the previous activation output coordinates is [3 4 7 11 12 14 16], and the offset of the output coordinates of the current filter kernel is [3 5 7 10 12 14 16]. By comparing the elements one, three, five, six, and seven are the same, and the rest are different, so the second mask is determined to be [0 1 0 1 0 0 0] and the third mask is [1 0 1 0 1 1 1]. The values ​​of the elements at the same position in the second and third masks are different.

[0087] Assign an index to the newly activated output coordinates and store the newly activated output coordinates and their offsets in global memory for subsequent access.

[0088] Through the first mask, the second mask, the third mask and the mask screening function, the mapping relationship between the index of the activated output coordinate corresponding to the current filter kernel and the index of the input coordinate is determined, that is, the rule table corresponding to the current filter kernel is obtained.

[0089] Since the performance of TPU global memory access is affected by the length of a single memory access data, in order to avoid a large number of memory access operations that lead to reduced efficiency of sparse convolution operations implemented on the TPU, the rule table is generated by replacing the memory access-based mask filtering with matrix multiplication-based mask filtering, which greatly improves the efficiency of rule table generation.

[0090] The method for generating a sparse convolution rule table provided in this embodiment implements sparse convolution operations on the TPU. In neural network applications, the TPU, with its systolic array structure, offers superior computing performance compared to GPUs, improving the efficiency of convolution operations. In order to implement sparse convolution operations on TPUs, a rule table generation method is provided, which specifically comprises the following steps: calculating the output coordinates of the input coordinates under each filter kernel and judging whether the calculated output coordinates are valid, and obtaining the first mask corresponding to each filter kernel based on the judgment result; then calculating the offset of each output coordinate for subsequent offset comparison; first generating a rule table corresponding to the first filter kernel, and establishing a mapping relationship between the index of the activated output coordinate (valid output coordinate) corresponding to the first filter kernel and the index of the input coordinate based on the first mask and the mask filtering function; then, for subsequent filter kernels, by comparing the offset of the output coordinate corresponding to the filter kernel with the offset of the activated output coordinate corresponding to each previous filter kernel, the activated output coordinate of the filter kernel is divided into newly activated output coordinates and existing output coordinates; and then, by using the mask generated in the comparison process, the first mask, and the mask filtering function, a mapping relationship between the index of the activated output coordinate (valid output coordinate) corresponding to the filter kernel and the index of the input coordinate is established to obtain a rule table, thereby implementing sparse convolution operations based on the rule table. The mask and mask filtering functions are used to implement sparse convolution operations on the TPU, effectively reducing the number of data memory accesses. This makes it more suitable for implementing sparse convolution operations on the TPU and improves computing efficiency.

[0091] Optionally, judging whether the output coordinates corresponding to the filter kernel are valid based on a preset judgment condition includes:

[0092] For each output coordinate corresponding to the filter kernel, determine whether the output coordinate is within the spatial range corresponding to the filter kernel, and determine whether the first intermediate coordinate of the output coordinate can divide the stride. The first intermediate coordinate of the output coordinate is equal to the input coordinate corresponding to the output coordinate plus the constant vector corresponding to the filter kernel. The constant vector corresponding to the filter kernel is the difference vector obtained by subtracting the offset vector of the filter kernel from the supplementary vector. If both judgment results are yes, the output coordinate is valid.

[0093] The stride is the distance between the positions of the filter kernel in the input data. A larger stride means less calculation and computing time.

[0094] Specifically, it is possible to determine whether the values ​​of the output coordinates in each dimension (such as the x-axis, y-axis, and z-axis) are between the upper and lower limits (usually 0) of the filter kernel in each dimension. If so, the output coordinates are within the spatial range corresponding to the filter kernel.

[0095] When making a spatial range judgment, in order to improve the accuracy of the judgment, multiple judgments may be made, such as 2 times, 3 times or other times.

[0096] The first intermediate coordinate is the coordinate obtained by adding the corresponding constant vector to the input coordinate. The output coordinate is obtained by dividing the first intermediate coordinate by the stride. If the first intermediate coordinate cannot divide the stride (usually 2), it means that the obtained output coordinate is invalid, that is, it is not an activated output coordinate.

[0097] By setting the spatial range and divisibility conditions, the calculated output coordinates are effectively judged, which realizes the reliable screening of the output coordinates, reduces the amount of output coordinate data required for subsequent processing, and improves the efficiency of rule table generation.

[0098] Optionally, determining whether the output coordinates are within a spatial range corresponding to the filter kernel includes:

[0099] For each dimension of the output coordinate, determine at least twice whether the coordinate value of the output coordinate in the dimension is greater than or equal to 0 and less than or equal to the upper boundary of the spatial range corresponding to the filter kernel in the dimension; if the results of at least two judgments in each dimension are yes, then the output coordinate is within the spatial range corresponding to the filter kernel.

[0100] The output coordinates are three-dimensional coordinates, corresponding to the x-axis, y-axis, and z-axis. The lower limit of the coordinate value of each dimension in the spatial range corresponding to the filter kernel is 0.

[0101] For each dimension of the output coordinate, repeatedly check whether the coordinate value lies between [0, max], where max is the upper bound of the coordinate value for that dimension. If it does, the corresponding mask element at that coordinate position is set to 1; otherwise, it is set to 0. Thus, each of the three dimensions corresponds to two masks consisting of 1s and 0s. An AND operation is performed on the six masks to obtain the first sub-mask. The output coordinates corresponding to the 0 positions in the first sub-mask are invalid coordinates.

[0102] Calculations related to the first sub-mask can be performed based on the INT8 (8-bit integer) computing unit in the TPU to save computing power.

[0103] Since division is not friendly to hardware devices, when judging the conditions related to integer division, you can first calculate the reciprocal of the stride, such as a 32-bit floating-point number, and replace the division by multiplying the first intermediate coordinate by the reciprocal of the stride.

[0104] Optionally, determining whether the first intermediate coordinate of the output coordinate can divide the stride includes:

[0105] Calculate the floating-point number corresponding to the reciprocal of the stride; convert the first intermediate coordinate of the output coordinate into a floating-point number and multiply it by the floating-point number corresponding to the reciprocal of the stride, convert the multiplication result into an integer to obtain a second intermediate coordinate; multiply the second intermediate coordinate by the stride to obtain a third intermediate coordinate; determine whether the first intermediate coordinate is consistent with the third intermediate coordinate; if they are consistent, the first intermediate coordinate of the output coordinate can divide the stride.

[0106] The floating point number corresponding to the reciprocal of the stride may be a 32-bit floating point number. Accordingly, the first intermediate coordinate of the output coordinate is converted into a 32-bit floating point number.

[0107] Multiply the 32-bit floating-point number converted from the first intermediate coordinate by the inverse of the stride (32 floating-point number), round the result of the multiplication to INT16 (16-bit integer) to obtain the second intermediate coordinate (INT16). Multiply the second intermediate coordinate by the stride to obtain the third intermediate coordinate. If the third intermediate coordinate is equal to the first intermediate coordinate, the element at the corresponding position of the output coordinate in the second sub-mask is set to 1; otherwise, it is set to 0. Traverse each output coordinate corresponding to the filter kernel to obtain the second sub-mask corresponding to the filter kernel. The output coordinates corresponding to the elements of the second sub-mask with a value of 0 are invalid output coordinates.

[0108] By performing an AND operation on a group of first sub-masks and second sub-masks corresponding to the same filter kernel, the first mask corresponding to the filter kernel is obtained.

[0109] Figure 6 A flow chart of a method for generating a sparse convolution rule table provided in another embodiment of the present application is provided in this embodiment. Figure 2 Based on the embodiment shown, step S204 and step S205 are further refined, as shown in FIG. Figure 6 As shown, the above steps S204 and S205 may include the following steps:

[0110] Step S401: Generate an input index sequence based on the number of input coordinates.

[0111] The input index array includes the index of each input coordinate.

[0112] The index sequence includes multiple indices, and the number of indices is equal to the number of offsets of the input coordinates of all input data (including input data with values ​​of 0 and non-zero).

[0113] For example, taking 15470 input coordinates as an example, the input index sequence is an arithmetic progression starting from 0, with a step size of 1, and a total of 15470 numbers.

[0114] Step S402: add an index to the activation output coordinates corresponding to the first filter kernel.

[0115] The first filter kernel is the first filter kernel visited when traversing the filter kernels of the convolutional neural network.

[0116] The activated output coordinates are used to determine the valid output coordinates. The index of the activated output coordinates starts from 0 and the step size is 1.

[0117] You can use the first mask corresponding to the filter kernel as the mask and the output coordinates corresponding to the filter kernel as the input to obtain the activation output coordinates corresponding to the filter kernel through the mask filtering function, and add an index to each activation output coordinate. Different activation output coordinates have different indices.

[0118] The index of the next activated output coordinate can be obtained by adding 1 to the index of the previous activated output coordinate.

[0119] Taking the first filter kernel as an example, if there are 10 activation output coordinates corresponding to the filter kernel, the indexes of the 10 activation output coordinates are values ​​from 0 to 9.

[0120] Step S403 : Using the first mask corresponding to the first filter kernel as a mask and the input index sequence as input, based on a mask filtering function, obtain the index of the input coordinate corresponding to the first filter kernel.

[0121] Step S404 : Establish a mapping relationship between the index of the activated output coordinate corresponding to the first filter core and the index of the input coordinate corresponding to the first filter core, and obtain a rule table corresponding to the first filter core.

[0122] By using a mask filtering function based on the first mask as a mask, a mapping relationship between the index of the activated output coordinate corresponding to the first filter kernel and the index of the input coordinate is obtained, and a rule table corresponding to the first filter kernel is obtained.

[0123] Step S405: For each subsequent filter kernel, based on the comparison result of the offset of the output coordinate corresponding to the filter kernel and the offset of the activated output coordinate corresponding to the previous filter kernel and the first mask corresponding to the filter kernel, generate the second mask and the third mask corresponding to the filter kernel.

[0124] The previous filter kernel is one or more filter kernels traversed before the current filter kernel. The second mask is used to describe the newly activated output coordinates of the filter kernel compared to the previous filter kernel, and the third mask is used to describe the activated output coordinates of the filter kernel that are the same as the previous filter kernel.

[0125] The newly activated output coordinates and their offsets can be stored in global memory. During comparison, the offsets of the stored activated output coordinates are read from global memory to local memory to obtain the offsets of the activated output coordinates corresponding to the previous filter kernel. The offsets of the output coordinates corresponding to the filter kernel are compared with the offsets of the activated output coordinates corresponding to the previous filter kernel in the local memory to obtain a comparison result, and based on the comparison result, a second mask and a third mask corresponding to the filter kernel are generated.

[0126] Optionally, the first mask, the second mask and the third mask are all 0-1 masks, the output coordinates corresponding to the elements with a value of 1 in the first mask are activated output coordinates, the output coordinates corresponding to the elements with a value of 1 in the second mask are newly activated output coordinates; the output coordinates corresponding to the elements with a value of 1 in the third mask are the same as the activated output coordinates corresponding to the previous filter kernel.

[0127] Optionally, generating a second mask and a third mask corresponding to the filter kernel based on a comparison result of the offset of the output coordinate corresponding to the filter kernel and the offset of the activated output coordinate corresponding to the previous filter kernel and the first mask corresponding to the filter kernel includes:

[0128] Based on the comparison result of the output coordinate offset corresponding to the filter kernel and the activated output coordinate offset corresponding to the previous filter kernel, a first mask matrix is ​​obtained; the first mask matrix is ​​left-multiplied by a preset sequence to obtain a fifth mask, and the values ​​of the elements greater than 0 in the fifth mask are converted to 1 to obtain the third mask; the first mask corresponding to the filter kernel and the third mask corresponding to the filter kernel are ANDed to obtain a fourth mask corresponding to the filter kernel; and the second mask corresponding to the filter kernel is determined to be the first mask corresponding to the filter kernel minus the fourth mask corresponding to the filter kernel.

[0129] Exemplarily, the preset sequence may be an arithmetic sequence, such as an arithmetic sequence starting from 1 and having a common difference of 1.

[0130] Figure 7 For this application Figure 6 The schematic diagram of the calculation process of the third mask and the fourth mask in the embodiment shown is as follows: Figure 7As shown in the figure, for example, the output coordinate offsets corresponding to the previous filter kernels are [5, 6, 11, 14, 16], and the output coordinate offsets corresponding to the current filter kernel are [5, 7, 11, 12, 16]. Using the output coordinate offsets corresponding to the previous filter kernels as column vectors and the output coordinate offsets corresponding to the current filter kernel as row vectors, a first mask matrix (0-1 mask matrix) is obtained through element-by-element comparison, where 0-1 indicates that the element value is either 0 or 1. This first mask matrix is ​​then multiplied on the left by a preset sequence, such as [1, 2, 3, 4, 5], to obtain the fifth mask corresponding to the current filter kernel, namely [1, 0, 3, 0, 5]. The output coordinates at the non-zero elements in the fifth mask are one of the existing active output coordinates. Elements greater than 0 in the fifth mask, namely 1, 3, and 5, are converted to 1, resulting in a 0-1 mask, [1, 0, 1, 0, 1], which is the third mask.

[0131] Continue to see Figure 7 , the first mask corresponding to the current filter kernel is [1,1,0,0,1], and the fourth mask is the mask obtained by the AND operation of the third mask and the first mask, that is, [1,0,0,0,1].

[0132] Because the third mask is derived by comparing offsets, the output coordinate at position 1 in the third mask is theoretically an existing active output coordinate. To improve the accuracy of the second mask, the first and third masks are first ANDed together. Ideally, the result of this AND operation is the third mask. The second mask is then subtracted from the first mask to obtain the second mask. This is equivalent to removing the output coordinates corresponding to the filter kernel that are identical to the previously activated output coordinates. As a result, the output coordinate at position 1 in the second mask is the newly activated output coordinate for the filter kernel. The fourth mask = first mask & third mask, and the second mask = first mask - (first mask & third mask), where "&" represents the AND operation.

[0133] The second mask and the third mask are obtained by the above method, which are used to represent the newly added activation output coordinates and the existing activation output coordinates in the current filter kernel respectively. The second mask and the third mask are highly accurate and provide reliable masks for the subsequent mask screening function, thereby improving the accuracy of rule table generation.

[0134] In some embodiments, the newly added activation output coordinates of the filter core and their offsets may also be stored in the global memory, so that the activation output coordinates stored in the global memory can be accessed when a subsequent filter core rule table is generated.

[0135] The newly added activation output coordinates of the filter core can be obtained by using the second mask corresponding to the filter core as the mask and the output coordinates corresponding to the filter core as the input through the TPU mask filtering function.

[0136] Optionally, the method further includes:

[0137] The activation output coordinates and offsets corresponding to the first filter kernel, as well as the newly activated output coordinates and offsets corresponding to the subsequent filter kernels, are stored in global memory; the offsets of the activation output coordinates corresponding to the previous filter kernel are read from global memory to local memory for offset comparison.

[0138] Step S406 : Using the fourth mask corresponding to the filter core as a mask and the output coordinates corresponding to the filter core as input, based on a mask screening function, obtain the newly added activated output coordinates of the filter core and assign an index.

[0139] The fourth mask corresponding to the filter kernel is a mask obtained by performing an AND operation on the first mask and the third mask corresponding to the filter kernel.

[0140] Step S407 , using the fourth mask corresponding to the filter core as a mask and the input index sequence as input, based on a mask screening function, obtains a mapping relationship between the index of the newly added activated output coordinate of the filter core and the index of the input coordinate.

[0141] In step S408, the second mask corresponding to the filter kernel is used as a mask, and the output coordinates corresponding to the filter kernel and the input index sequence are used as inputs. Based on the mask screening function, a mapping relationship between the index of the activated output coordinates of the filter kernel and the index of the input coordinates that are the same as those of the previous filter kernel is obtained.

[0142] The mapping relationship between the index of the newly added activated output coordinates of the filter core and the index of the input coordinates, and the mapping relationship between the index of the activated output coordinates of the filter core and the index of the input coordinates that are the same as the previous filter core, constitute the rule table corresponding to the filter core.

[0143] Traverse each filter core and obtain the rule table corresponding to each filter core through the above steps.

[0144] In this embodiment, a mask matrix is ​​obtained by comparing the offsets, and a third mask and a second mask are obtained by a series of operations on the mask matrix. A rule table is obtained based on the first to third masks and the mask screening function. This does not require excessive data memory access, and the calculation operation is simple. It can better adapt to the computing characteristics and memory access characteristics of the TPU, thereby improving the efficiency of the sparse convolution operation implemented on the TPU.

[0145] In the rule table obtained using the aforementioned method, the output coordinate indices are not sorted by the offset of the output coordinates, but rather by the order in which the newly activated output coordinates appear. Therefore, after obtaining the rule table corresponding to each filter kernel, the active output coordinates in the rule table can be re-indexed in ascending order of the output coordinate offset, so that the indices of the active output coordinates with larger offsets in the rule table are higher than those with smaller offsets.

[0146] In order to achieve index sorting by offset, the offsets of the activation output coordinates corresponding to the filter core can be sorted first, and the offsets of the activation output coordinates corresponding to the filter core before and after sorting can be compared to obtain a 0-1 mask matrix. Through the 0-1 mask matrix, a mask is obtained that characterizes the order of the index of the activation output coordinates in the new rule table. Based on the data aggregation function of the TPU, the index of the original output coordinate in the rule table is replaced with the new index after sorting indicated in the mask. The indexes in the rule table can also be rearranged in order from small to large.

[0147] Optionally, the method further includes:

[0148] For each filter kernel, the offsets of the activated output coordinates corresponding to the filter kernel are sorted in ascending order of the offsets; based on the position changes of the offsets of the output coordinates before and after the sorting, a second mask matrix corresponding to the filter kernel is generated; based on the second mask matrix corresponding to the filter kernel, the index of the activated output coordinates in the rule table corresponding to the filter kernel is updated, and the rule table corresponding to the filter kernel is rearranged in ascending order of the index of the activated output coordinates.

[0149] The second mask matrix is ​​a 0-1 matrix, with the vector corresponding to the offset after sorting as the column vector and the vector corresponding to the offset before sorting as the row vector. The offsets before and after sorting are compared element by element to see if they are the same. If the i-th offset in the column vector is the same as the j-th offset in the row vector, then the value of the element in the i-th row and j-th column of the second mask matrix is ​​1, otherwise it is 0.

[0150] Specifically, a new serial number of the index of the activated output coordinate may be determined based on the second mask matrix, and based on the new serial number, the order of the index of the activated output coordinate in the rule table may be adjusted to achieve rearrangement of the rule table.

[0151] Optionally, updating the index of the activated output coordinate in the rule table corresponding to the filter kernel based on the second mask matrix corresponding to the filter kernel includes:

[0152] The second mask matrix corresponding to the filter core is multiplied on the left by a preset continuous sequence to obtain a sequence mapping array; based on the value of each element of the sequence mapping array, the index of the activated output coordinate in the rule table corresponding to the filter core is updated.

[0153] The preset continuous sequence may be an arithmetic sequence starting from 0, with a tolerance of 1 and a length equal to the number of activated output coordinates corresponding to the filter kernel.

[0154] After obtaining the second mask matrix corresponding to the filter kernel, a preset continuous sequence is constructed and the second mask matrix is ​​left-multiplied by the preset continuous sequence to obtain a sequence mapping array. Each element in the sequence mapping array is used to indicate the new sequence number of the index of the activation output coordinate at the corresponding position. Then, using the data aggregation function of the TPU, the order of the indexes in the rule table is adjusted according to the order indicated by the sequence mapping array to obtain a new rule table.

[0155] For example, Figure 8 A schematic diagram of a process for generating a sequence number mapping array is provided for one embodiment of the present application, as shown in FIG. Figure 8 As shown, the vector corresponding to the offset of the output coordinate corresponding to the filter kernel is [16, 6, 11, 14, 5]. The vector corresponding to the offset after sorting is [5, 6, 11, 14, 16]. Using the vector corresponding to the offset before sorting as the column vector and the vector corresponding to the offset after sorting as the row vector, we compare the elements one by one to obtain the second mask matrix. This second mask matrix is ​​then multiplied on the left by the arithmetic progression [1, 2, 3, 4, 5] (i.e., a preset continuous sequence), resulting in a sequence mapping array, [5, 2, 3, 4, 1]. This means that the original first index in the rule table has a new sequence number of 5, i.e., the fifth index, and the original fifth index has a new sequence number of 1, i.e., the first index. Based on the data aggregation function of TPU, the original index in the rule table is replaced with the sorted index according to the sequence mask. Taking the index of the activated output coordinate in the rule table as [0 1 2 3 4] as an example, the serial number mapping array is adjusted to [4 1 2 3 0] to sort the index of the output coordinate in the rule table according to the offset.

[0156] Furthermore, after obtaining the rule table corresponding to each filter core, it also includes:

[0157] Based on the index of the input coordinates in the rule table corresponding to each filter core, the activation input data is extracted from the input data and aggregated to obtain the activation input matrix corresponding to each filter core; the activation input matrix corresponding to each filter core is multiplied by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; based on the rule table corresponding to each filter core, the activation output data corresponding to each filter core is scattered to the corresponding output coordinates, and the output data under the same output coordinates are superimposed to obtain the sparse convolution operation result.

[0158] After obtaining multiple activation output data of the filter kernel, the multiple activation output data are scattered or dispersed to the corresponding output coordinates based on the index of the output coordinate corresponding to each activation data in the rule table. The output coordinate corresponding to the activation output data is the output coordinate corresponding to the input coordinate of the input data used to calculate the activation output data. When the activation output data of different filter kernels correspond to the same output coordinate index, the activation output data of different filter kernels are superimposed to obtain the final activation output data at the output coordinate. After the operation is completed, the result of the sparse convolution operation is obtained, that is, the output tensor.

[0159] Furthermore, the output tensor can be sent to downstream devices or modules for further analysis, such as target recognition based on the output tensor.

[0160] After obtaining the rule table, the sparse convolution operation results are obtained by aggregating the input data, scattering the output data, and superimposing it. The sparse convolution operation is implemented on the TPU, which improves the efficiency of the sparse convolution operation and reduces the operation cost.

[0161] The steps provided in the various method embodiments of the present application can be executed in parallel or in series if logic permits. The present application does not limit the order in which the steps are executed, and the order shown in the drawings is only an example.

[0162] This embodiment of the present application provides a sparse convolution operation method, including:

[0163] Based on the index of the input coordinates in the rule table corresponding to each filter core, the activation input data is extracted from the input data and aggregated to obtain the activation input matrix corresponding to each filter core; the activation input matrix corresponding to each filter core is multiplied by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; based on the rule table corresponding to each filter core, the activation output data corresponding to each filter core is scattered to the corresponding output coordinates, and the output data at the same output coordinates are superimposed to obtain the sparse convolution operation result. Wherein, the rule table is obtained based on the sparse convolution rule table generation method provided in any embodiment of the present application.

[0164] Figure 9A schematic diagram of a sparse convolution operation process provided in one embodiment of the present application is shown in FIG. Figure 9 As shown in FIG, the sparse convolution operation process includes two stages: the rule table generation stage and the convolution operation stage.

[0165] In the rule table generation stage, for each filter core, the coordinates of the output data corresponding to the filter core (i.e., output coordinates) are first calculated based on the coordinates of the input data (i.e., input coordinates). Specifically, the coordinates of the output data can be calculated based on the addition calculation unit (Add) and the subtraction calculation unit (Sub). Then, the CMP (Chip multiprocessors) instruction is used to check the validity of each calculated output coordinate to determine whether each output coordinate is valid, thereby obtaining the activated output coordinates or valid output coordinates corresponding to the filter core. The offset of the activated output coordinate corresponding to the filter core is calculated, and the offset of the confirmed activated output coordinates is updated based on the offset of the activated output coordinates corresponding to the filter core in each round of traversal. The confirmed activated output coordinates include the newly added activated output coordinates corresponding to each filter core. The offset of the activated output coordinate corresponding to the currently traversed filter core is compared with the offset of the confirmed activated output coordinate to obtain a 0-1 mask. Based on the 0-1 mask, the rule table corresponding to the filter core is generated using a mask calculation method. The specific process can be found in the above embodiment.

[0166] In the convolution operation stage, valid input data is input, and the input data that needs to participate in the calculation is aggregated through the index of the input coordinate in the rule table, that is, the input data is activated; the aggregated input data is multiplied by the weight matrix of the filter kernel to obtain the output data; based on the index of the output coordinate in the rule table, the output data is scattered and superimposed to the specified position, that is, the position of the corresponding output coordinate.

[0167] Figure 10 A schematic diagram of a device for generating a sparse convolution rule table according to an embodiment of the present application, wherein the device for generating a sparse convolution rule table is applied to a tensor processing unit, such as Figure 10 As shown, the sparse convolution rule table generation device includes: an output coordinate calculation module 810, a first mask acquisition module 820, an offset calculation module 830, a first rule table generation module 840 and a second rule table generation module 850.

[0168] The output coordinate calculation module 810 is used to calculate the output coordinates corresponding to the input coordinates in each filter core; the first mask acquisition module 820 is used to determine, for each filter core, based on preset judgment conditions, whether the output coordinates corresponding to the filter core are valid, and obtain the first mask corresponding to the filter core based on the judgment result; the offset calculation module 830 is used to calculate the offset of each of the output coordinates; the first rule table generation module 840 is used to determine, for the first filter core, based on the first mask corresponding to the first filter core, the mapping relationship between the index of the input coordinate and the index of the activated output coordinate corresponding to the first filter core, and obtain the rule table corresponding to the first filter core; the second rule table generation module 850 is used to determine, for each subsequent filter core, based on the first mask corresponding to the filter core and the comparison result of the offset of the output coordinate corresponding to the filter core and the offset of the activated output coordinate corresponding to the previous filter core, the mapping relationship between the index of the input coordinate and the index of the activated output coordinate corresponding to the filter core, and obtain the rule table corresponding to the filter core; wherein, the activated output coordinate is the output coordinate judged to be valid.

[0169] Optionally, the first rule table generating module 840 is specifically configured to:

[0170] Based on the number of input coordinates, an input index sequence is generated, wherein the input index sequence includes the index of each input coordinate; an index is added to the activated output coordinate corresponding to the first filter core; using the first mask corresponding to the first filter core as a mask and the input index sequence as input, based on the mask filtering function, the index of the input coordinate corresponding to the first filter core is obtained; a mapping relationship between the index of the activated output coordinate corresponding to the first filter core and the index of the input coordinate corresponding to the first filter core is established to obtain a rule table corresponding to the first filter core.

[0171] Optionally, the second rule table generating module 850 includes:

[0172] The second and third mask generating units are used to generate the second mask and the third mask corresponding to the filter kernel based on the comparison result of the output coordinate offset corresponding to the filter kernel and the offset of the activation output coordinate corresponding to the previous filter kernel and the first mask corresponding to the filter kernel, wherein the second mask is used to describe the newly added activation output coordinates of the filter kernel compared with the previous filter kernel, and the third mask is used to describe the activation output coordinates of the filter kernel that are the same as the previous filter kernel; a new index assigning unit is added, which is used to use the fourth mask corresponding to the filter kernel as a mask and the output coordinates corresponding to the filter kernel as input, and based on the mask screening function, obtain the newly added activation output coordinates of the filter kernel and assign an index , wherein the fourth mask corresponding to the filter kernel is a mask obtained by performing an AND operation on the first mask and the third mask corresponding to the filter kernel; a new index mapping unit is added, which is used to use the fourth mask corresponding to the filter kernel as a mask, the input index sequence as input, and based on the mask screening function, obtain the mapping relationship between the index of the newly added activated output coordinate of the filter kernel and the index of the input coordinate; the existing coordinate index mapping unit is used to use the second mask corresponding to the filter kernel as a mask, the output coordinates corresponding to the filter kernel and the input index sequence as input, and based on the mask screening function, obtain the mapping relationship between the index of the activated output coordinate of the filter kernel that is the same as that of the previous filter kernel and the index of the input coordinate.

[0173] Optionally, the first mask, the second mask and the third mask are all 0-1 masks, the output coordinates corresponding to the elements with a value of 1 in the first mask are activated output coordinates, the output coordinates corresponding to the elements with a value of 1 in the second mask are newly activated output coordinates; the output coordinates corresponding to the elements with a value of 1 in the third mask are the same as the activated output coordinates corresponding to the previous filter kernel.

[0174] Accordingly, the second and third mask generating units are specifically configured to:

[0175] Based on the comparison result of the output coordinate offset corresponding to the filter kernel and the activated output coordinate offset corresponding to the previous filter kernel, a first mask matrix is ​​obtained; the first mask matrix is ​​left-multiplied by a preset sequence to obtain a fifth mask, and the values ​​of the elements greater than 0 in the fifth mask are converted to 1 to obtain the third mask; the first mask corresponding to the filter kernel and the third mask corresponding to the filter kernel are ANDed to obtain a fourth mask corresponding to the filter kernel; and the second mask corresponding to the filter kernel is determined to be the first mask corresponding to the filter kernel minus the fourth mask corresponding to the filter kernel.

[0176] Optionally, the device further includes:

[0177] The global storage module is used to store the activation output coordinates and their offsets corresponding to the first filter kernel, as well as the newly activated output coordinates and their offsets corresponding to subsequent filter kernels, into the global memory; the memory access module is used to read the offsets of the activation output coordinates corresponding to the previous filter kernel from the global memory to the local memory for offset comparison.

[0178] Optionally, the device further includes:

[0179] An offset sorting module is used to sort the offsets of the activated output coordinates corresponding to each filter core in order from small to large offsets; a second mask matrix generation module is used to generate a second mask matrix corresponding to the filter core based on the position changes of the offsets of the output coordinates before and after sorting; an index update module is used to update the index of the activated output coordinates in the rule table corresponding to the filter core based on the second mask matrix corresponding to the filter core.

[0180] Optional index update module, specifically used for:

[0181] The second mask matrix corresponding to the filter core is multiplied on the left by a preset continuous sequence to obtain a sequence mapping array; based on the value of each element of the sequence mapping array, the index of the activated output coordinate in the rule table corresponding to the filter core is updated.

[0182] Optionally, the first mask acquisition module 820 includes:

[0183] The first condition determination unit is used to determine, for each output coordinate corresponding to each filter, whether the output coordinate is within the spatial range corresponding to the filter kernel; the second condition determination unit is used to determine, for each output coordinate corresponding to each filter, whether the first intermediate coordinate of the output coordinate can divide the stride, the first intermediate coordinate of the output coordinate is equal to the input coordinate corresponding to the output coordinate plus the constant vector corresponding to the filter kernel, and the constant vector corresponding to the filter kernel is the difference vector obtained by subtracting the offset vector of the filter kernel from the supplementary vector; the first mask generation unit is used to generate the first mask corresponding to each filter kernel based on the judgment results output by the first condition determination unit and the second condition determination unit.

[0184] If the judgment results output by the first condition determination unit and the second condition determination unit are both yes, the corresponding output coordinates are valid, and the value of the element at the corresponding position in the first mask is 1.

[0185] Optionally, the first condition determination unit is specifically configured to:

[0186] For each output coordinate corresponding to each filter, for each dimension of the output coordinate, determine at least twice whether the coordinate value of the output coordinate in the dimension is greater than or equal to 0 and less than or equal to the upper boundary of the spatial range corresponding to the filter kernel in the dimension; if the results of at least two judgments in each dimension are yes, then the output coordinate is within the spatial range corresponding to the filter kernel.

[0187] Optionally, the second condition determination unit is specifically configured to:

[0188] Calculate the floating-point number corresponding to the reciprocal of the stride; for each output coordinate corresponding to each filter, convert the first intermediate coordinate of the output coordinate into a floating-point number and multiply it by the floating-point number corresponding to the reciprocal of the stride, and convert the multiplication result into an integer to obtain a second intermediate coordinate; multiply the second intermediate coordinate by the stride to obtain a third intermediate coordinate; determine whether the first intermediate coordinate is consistent with the third intermediate coordinate; if they are consistent, the first intermediate coordinate of the output coordinate can divide the stride.

[0189] The device for generating a sparse convolution rule table provided in an embodiment of the present application can execute the method for generating a sparse convolution rule table provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.

[0190] The present application also provides a sparse convolution operation device, which is applied to a tensor processing unit and includes:

[0191] An aggregation module is used to extract and aggregate activation input data from the input data based on the index of the input coordinates in the rule table corresponding to each filter core, so as to obtain the activation input matrix corresponding to each filter core; an output calculation module is used to multiply the activation input matrix corresponding to each filter core by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; a scattering and superposition module is used to scatter the activation output data corresponding to each filter core to the corresponding output coordinates based on the rule table corresponding to each filter core, and superimpose the output data under the same output coordinates to obtain the sparse convolution operation result; wherein, the rule table is generated based on the sparse convolution rule table generation method provided in any embodiment of the present application.

[0192] An embodiment of the present application also provides a tensor processing unit, which is used to perform sparse convolution operations and stores a sparse convolution rule table. The rule table is generated based on the sparse convolution rule table generation method provided in any embodiment of the present application.

[0193] The tensor processing unit is specifically used to:

[0194] Based on the index of the input coordinates in the rule table corresponding to each filter core, the activation input data is extracted from the input data and aggregated to obtain the activation input matrix corresponding to each filter core; the activation input matrix corresponding to each filter core is multiplied by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; based on the rule table corresponding to each filter core, the activation output data corresponding to each filter core is scattered to the corresponding output coordinates, and the output data under the same output coordinates are superimposed to obtain the sparse convolution operation result.

[0195] One embodiment of the present application provides a computer-readable storage medium having computer-executable instructions stored thereon. When a processor executes the computer-executable instructions, the method provided by any embodiment of the present application is implemented.

[0196] The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic tape, a floppy disk, or an optical data storage device.

[0197] One embodiment of the present application provides a computer program product, including a computer program, which implements the method provided in any embodiment of the present application when executed by a tensor processing unit.

[0198] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0199] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for generating a sparse convolution rule table, characterized in that: The method is applied to a tensor processing unit, and the method includes: Calculate the output coordinates corresponding to the input coordinates in each filter kernel; For each filter kernel, based on a preset determination condition, determining whether the output coordinates corresponding to the filter kernel are valid, and obtaining a first mask corresponding to the filter kernel based on the determination result; Calculating the offset of each of the output coordinates; For a first filter kernel, determining, based on a first mask corresponding to the first filter kernel, a mapping relationship between an index of an input coordinate and an index of an activated output coordinate corresponding to the first filter kernel, and obtaining a rule table corresponding to the first filter kernel; For each subsequent filter kernel, determining a mapping relationship between an index of the input coordinate and an index of the activated output coordinate corresponding to the filter kernel based on the first mask corresponding to the filter kernel and a comparison result of the offset of the output coordinate corresponding to the filter kernel and the offset of the activated output coordinate corresponding to the previous filter kernel, and obtaining a rule table corresponding to the filter kernel; The activated output coordinates are output coordinates that are determined to be valid.

2. The method according to claim 1, characterized in that Determining, based on a first mask corresponding to the first filter kernel, a mapping relationship between an index of an input coordinate and an index of an activated output coordinate corresponding to the first filter kernel, and obtaining a rule table corresponding to the first filter kernel, including: Based on the number of input coordinates, generate an input index sequence, the input index sequence including an index of each input coordinate; Add indices to the activation output coordinates corresponding to the first filter kernel; Using the first mask corresponding to the first filter kernel as the mask and the input index sequence as input, based on the mask filtering function, obtain the index of the input coordinate corresponding to the first filter kernel; A mapping relationship between the index of the activation output coordinate corresponding to the first filter kernel and the index of the input coordinate corresponding to the first filter kernel is established to obtain a rule table corresponding to the first filter kernel.

3. The method according to claim 2, characterized in that Determining a mapping relationship between an index of an input coordinate and an index of an activated output coordinate corresponding to the filter kernel based on a first mask corresponding to the filter kernel and a comparison result of an offset of an output coordinate corresponding to the filter kernel and an offset of an activated output coordinate corresponding to a previous filter kernel, including: Generate a second mask and a third mask corresponding to the filter kernel based on a comparison result of the output coordinates corresponding to the filter kernel and the offset of the activated output coordinates corresponding to the previous filter kernel and a first mask corresponding to the filter kernel, wherein the second mask is used to describe the activated output coordinates newly added by the filter kernel compared to the previous filter kernel, and the third mask is used to describe the activated output coordinates of the filter kernel that are the same as those of the previous filter kernel; Taking the fourth mask corresponding to the filter kernel as the mask and the output coordinates corresponding to the filter kernel as input, based on the mask screening function, obtain the newly added activated output coordinates of the filter kernel and assign them an index, wherein the fourth mask corresponding to the filter kernel is a mask obtained by performing an AND operation on the first mask and the third mask corresponding to the filter kernel; Using the fourth mask corresponding to the filter kernel as the mask and the input index sequence as input, based on the mask screening function, obtain a mapping relationship between the index of the newly activated output coordinate of the filter kernel and the index of the input coordinate; Taking the second mask corresponding to the filter kernel as the mask, and taking the output coordinates corresponding to the filter kernel and the input index sequence as input respectively, based on the mask screening function, a mapping relationship between the index of the activated output coordinate of the filter kernel and the index of the input coordinate that is the same as the previous filter kernel is obtained.

4. The method according to claim 3, characterized in that The first mask, the second mask, and the third mask are all 0-1 masks. The output coordinates corresponding to the elements with a value of 1 in the first mask are activated output coordinates, and the output coordinates corresponding to the elements with a value of 1 in the second mask are newly activated output coordinates. The output coordinates corresponding to the elements with a value of 1 in the third mask are the same as the activated output coordinates corresponding to the previous filter kernel. Generating a second mask and a third mask corresponding to the filter kernel based on a comparison result of the offset of the output coordinate corresponding to the filter kernel and the offset of the activated output coordinate corresponding to the previous filter kernel and the first mask corresponding to the filter kernel, including: Obtaining a first mask matrix based on a comparison result of an offset of the output coordinate corresponding to the filter kernel and an offset of an activated output coordinate corresponding to a previous filter kernel; Multiplying the first mask matrix by a preset sequence on the left to obtain a fifth mask, and converting the values ​​of elements greater than 0 in the fifth mask to 1 to obtain the third mask; Performing an AND operation on the first mask corresponding to the filter core and the third mask corresponding to the filter core to obtain a fourth mask corresponding to the filter core; A second mask corresponding to the filter core is determined to be the first mask corresponding to the filter core minus a fourth mask corresponding to the filter core.

5. The method according to claim 3, characterized in that The method further comprises: Store the activation output coordinates and offsets corresponding to the first filter kernel, as well as the newly activated output coordinates and offsets corresponding to subsequent filter kernels, into global memory; Read the offset of the activation output coordinates corresponding to the previous filter kernel from the global memory to the local memory for offset comparison.

6. The method according to any one of claims 2 to 5, characterized in that: The method further comprises: For each filter kernel, sort the offsets of the activation output coordinates corresponding to the filter kernel in ascending order of offset; generating a second mask matrix corresponding to the filter kernel based on a position change of the offset of the output coordinates before and after the sorting; Based on the second mask matrix corresponding to the filter kernel, the index of the activated output coordinate in the rule table corresponding to the filter kernel is updated.

7. The method according to claim 6, characterized in that Based on the second mask matrix corresponding to the filter kernel, updating the index of the activated output coordinate in the rule table corresponding to the filter kernel includes: Multiplying the filter kernel corresponding to the second mask matrix by a preset continuous sequence to obtain a sequence mapping array; Based on the values ​​of the elements of the sequence number mapping array, the index of the activated output coordinate in the rule table corresponding to the filter core is updated.

8. The method according to any one of claims 1 to 5, characterized in that Based on a preset determination condition, determining whether the output coordinates corresponding to the filter kernel are valid includes: For each output coordinate corresponding to the filter kernel, determine whether the output coordinate is within the spatial range corresponding to the filter kernel, and determine whether the first intermediate coordinate of the output coordinate can divide the stride, where the first intermediate coordinate of the output coordinate is equal to the input coordinate corresponding to the output coordinate plus the constant vector corresponding to the filter kernel, where the constant vector corresponding to the filter kernel is a difference vector obtained by subtracting the offset vector of the filter kernel from the supplementary vector; If both judgment results are yes, the output coordinates are valid.

9. The method according to claim 8, characterized in that Determining whether the output coordinates are within the spatial range corresponding to the filter kernel includes: For each dimension of the output coordinate, determine at least twice whether the coordinate value of the output coordinate in the dimension is greater than or equal to 0 and less than or equal to the upper boundary of the spatial range corresponding to the filter kernel in the dimension; If the results of at least two judgments in each dimension are both yes, then the output coordinates are within the spatial range corresponding to the filter kernel.

10. The method according to claim 8, characterized in that Determining whether the first intermediate coordinate of the output coordinate can divide the stride evenly includes: Calculate the floating point number corresponding to the reciprocal of the stride; Converting the first intermediate coordinate of the output coordinate into a floating point number and multiplying it by the floating point number corresponding to the reciprocal of the stride, and converting the multiplication result into an integer to obtain a second intermediate coordinate; Multiply the second intermediate coordinate by the stride to obtain a third intermediate coordinate; Determining whether the first intermediate coordinate is consistent with the third intermediate coordinate; If they are consistent, the first intermediate coordinate of the output coordinate can divide the stride.

11. A sparse convolution operation method, characterized in that: The method is applied to a tensor processing unit, and the method includes: Based on the index of the input coordinates in the rule table corresponding to each filter core, the activation input data is extracted from the input data and aggregated to obtain the activation input matrix corresponding to each filter core; Multiply the activation input matrix corresponding to each filter core by the weight matrix of the corresponding filter core to obtain the activation output data corresponding to each filter core; Based on the rule table corresponding to each filter core, the activation output data corresponding to each filter core is scattered to the corresponding output coordinate, and the output data under the same output coordinate is superimposed to obtain the sparse convolution operation result; Wherein, the rule table is generated based on the method provided in any one of claims 1-10.

12. A tensor processing unit for performing sparse convolution operations, characterized in that: A sparse convolution rule table is stored thereon, and the rule table is generated based on the method provided in any one of claims 1 to 10.

Images