Optimization method and optimization system of SpMM operator and computing equipment

Abstract

The invention discloses an optimization method and an optimization system of an SpMM operator, and computing equipment. After the optimization system receives the SpMM operator, the kernel of the SpMM operator can be segmented by using a plurality of segmentation modes to obtain a plurality of segmentation results. And the optimization system screens the slicing results by using constraint conditions related to the characteristics of the SpMM operator operated by equipment for operating the SpMM operator, and screens out several segmentation results adaptive to the constraint conditions, so that the SpMM operator operates according to the screened segmentation results in the reasoning process, and the calculation performance of the SpMM operator is improved. In addition, the number of slice results screened out by an optimization system by utilizing constraint conditions may be large, so that the search overhead during SpMM operator operation is large. The optimization system can select the concentrated slicing result with the highest performance based on the equipment performance for running the slicing result, so that the SpMM operator runs according to the screened slicing result in the reasoning process, and the calculation performance of the SpMM operator can be further improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an optimization method, an optimization system and a computing device for a SpMM operator. Background Art

[0002] With the development of artificial intelligence (AI) technology, neural network models can be applied in various fields, such as computer vision, natural language processing, speech recognition, reinforcement learning, medical diagnosis, finance, etc. As neural network models continue to learn in depth, the scale of neural network models becomes larger and larger. Large-scale neural network models require a lot of computing resources and storage resources, which leads to longer delays in reasoning of neural network models. Therefore, how to reduce the reasoning delay of neural network models is an urgent problem to be solved. Summary of the invention

[0003] In order to solve the above problems, an optimization method for a SpMM operator is provided in an embodiment of the present application. After obtaining the SpMM operator, the kernel of the SpMM operator is randomly segmented to obtain a variety of slicing results, and the slicing results that are suitable for the constraints are screened out using the constraints related to the characteristics of the device running the SpMM operator. During the reasoning process of the SpMM operator, it can be run based on the screened slicing results to improve the computing performance of the SpMM operator. In addition, the present application also provides an optimization method for the SpMM operator corresponding to the optimization method of the SpMM operator.

[0004] To this end, the following technical solutions are adopted in the embodiments of the present application:

[0005] In a first aspect, an embodiment of the present application provides an optimization method for a SpMM operator, the method comprising: obtaining a first SpMM operator from a neural network; slicing the kernel of the first SpMM operator by a plurality of slicing methods to obtain a plurality of slicing results, the slicing results comprising the number of slices obtained by slicing the kernel of the first SpMM operator and the dimensions of the slices; screening the plurality of slicing results by using constraints to screen out at least one slicing result that is adapted to the constraints, the constraints being related to the characteristics of the device running the first SpMM operator.

[0006] In this implementation, the optimization method is generally performed by the optimization system. After receiving the SpMM operator, the optimization system can use multiple slicing methods to slice the kernel of the SpMM operator to obtain multiple slicing results. The optimization system then uses the constraint conditions related to the characteristics of the device running the SpMM operator to filter the slicing results, and filters out several slicing results that are suitable for the constraint conditions, so that the SpMM operator runs according to the filtered slicing results during the reasoning process, thereby improving the computing performance of the SpMM operator.

[0007] In one embodiment, the method further includes: when the number of the at least one slicing result is greater than a preset number, based on the device performance of the first SpMM operator that runs the slicing result, filtering out the preset number of slicing results from the at least one slicing result.

[0008] In this implementation, the number of slice results screened out by the optimization system using the constraint conditions may be relatively large, resulting in a relatively large search overhead when the SpMM operator is running. The optimization system can select the concentrated slice result with the highest performance based on the performance of the device running the slice result, so that the SpMM operator runs according to the screened segmentation results during the inference process, which can further improve the computing performance of the SpMM operator.

[0009] In one implementation, the dimensions of different slices in the slicing results are all the same, and the dimension of the slice refers to the number of rows and columns of the matrix corresponding to the slice.

[0010] In this implementation, in the process of optimizing the system to slice the kernel of the SpMM operator, the dimensions of multiple slices in each slicing result are made the same, which can improve the balance of the load of running each slice.

[0011] In one embodiment, before using the constraint condition to filter the multiple slicing results and filter out at least one slicing result that is adapted to the constraint condition, it also includes: when the integer multiple of the dimension of the slices in the slicing result is different from the dimension of the first SpMM operator, adding zero elements to the slices with missing elements in the slicing result.

[0012] In this implementation, when the priority system determines that the dimension of the kernel of the SpMM operator is not the same as the integer multiple of the dimension of the slice, the dimension of some slices will be smaller than the dimension of other slices. In order to keep the dimensions of all slices in the slicing result the same, zero elements are added to the slices with smaller dimensions, so that the dimensions of all slices in the slicing result are the same, so as to balance the load of each slice.

[0013] In one embodiment, the constraint condition includes a register resource constraint condition, and the register resource constraint condition is related to the maximum memory of the register accessed when the device runs the slice; the constraint condition is used to filter the multiple slice results to filter out at least one slice result that is suitable for the constraint condition, specifically including: based on the dimension of the slice in the slice result, determining the memory occupied by the resources required for the device to run the slice; if the memory occupied by the resources required for the device to run the slice exceeds the maximum memory of the register accessed when the device runs the slice, discarding the slice result corresponding to the slice.

[0014] In this implementation, if the memory of the register accessed by the device corresponding to the running slice is larger than the maximum memory of the register configured by itself, it is necessary to access the local memory, resulting in reduced performance of the slice. Therefore, the optimization system can delete the slice results corresponding to the slices with larger access memory and filter the slice results with lower performance, which can improve the computing performance of the SpMM operator.

[0015] In one embodiment, the constraint condition also includes a hardware utilization constraint condition, which is related to the number of cores of the device running the first SpMM operator; the utilization constraint condition is used to filter the multiple slicing results to select at least one slicing result that is suitable for the constraint condition, specifically including: when the number of slices of the slicing result is less than the number of cores of the device running the first SpMM operator, the slicing result is discarded.

[0016] In this implementation, the device running the SpMM operator generally has multiple cores, such as a GPU, and generally includes multiple stream processors, each of which processes one slice. When the device running the SpMM operator runs multiple cores in parallel to perform computing tasks, if the number of slices in the slicing result is less than the number of cores, some cores will be idle, resulting in the hardware resources of the device running the SpMM operator not being fully utilized. Therefore, the optimization system can delete the slicing results where the number of slices is less than the number of cores, which can improve the computing performance of the SpMM operator.

[0017] In one embodiment, the constraint condition also includes a load balancing constraint condition, and the load balancing constraint condition is related to a slice balance threshold, and the slice balance threshold refers to the difference between the maximum proportion of non-zero elements in the slice in the slice result and the minimum proportion of non-zero elements in the slice in the slice result; the use of the constraint condition to filter the multiple slice results to filter out at least one slice result that is suitable for the constraint condition specifically includes: when the difference between the proportion of non-zero elements in the first slice and the proportion of non-zero elements in the second slice in the slice result is greater than the slice balance threshold, the slice result is discarded.

[0018] In this embodiment, each slice may include zero elements. The fewer zero elements in a slice, the more times the device running the slice accesses the memory. The more zero elements in a slice, the more times the device running the slice accesses the memory. If the difference between the zero elements of multiple slices in a slicing result is relatively large, the difference in the number of times the device accesses the memory when running the slice is relatively large, resulting in an extremely unbalanced load on the slicing result. Therefore, the optimization system can delete the slicing results in which the difference between the zero elements of multiple slices in the slicing result is relatively large, which can improve the computing performance of the SpMM operator.

[0019] In one embodiment, the kernel of the first SpMM operator is sliced ​​by multiple slicing methods to obtain multiple slicing results, specifically including: reordering the elements of each row in the sparse matrix based on the number of non-zero elements in each row in the sparse matrix calculated by the first SpMM operator; and sequentially allocating the elements of the output matrix calculated from the reordered row of elements to the slice with the least non-zero elements.

[0020] In this embodiment, since there are a large number of zero elements in the sparse matrix, the zero elements in the sparse matrix and the elements of the dense matrix may generate zero elements by matrix multiplication, which will affect the number of zero elements in the slice. Before the optimization system uses the load balancing constraint to filter the low-performance slices, it first sorts the elements of each row of the sparse matrix according to the number of non-zero elements. Then, the optimization system distributes the elements of each row to the slice with the least non-zero elements in turn according to the order of arrangement, so that the non-zero elements between different slices are kept close, reducing the impact of the zero elements in the sparse matrix on the load balancing constraint, thereby achieving balanced distribution of computing load.

[0021] In one embodiment, the method further includes: compiling the filtered slicing results to obtain a code corresponding to the slicing results, and when the code receives a sparse matrix and a dense matrix, slicing the sparse matrix and the dense matrix, and performing matrix multiplication to obtain an output matrix.

[0022] In this embodiment, after the optimization system screens out a set number of high-performance slicing results, code generation, code compilation and code execution are performed on each slicing result to obtain running code, so that the SpMM operator receives sparse matrices and dense matrices, performs matrix multiplication, and realizes the function of the SpMM operator.

[0023] In a second aspect, an optimization system provided by an embodiment of the present application includes: a transceiver unit, used to obtain a first SpMM operator from a neural network; a processing unit, used to slice the kernel of the first SpMM operator in a plurality of slice modes, respectively, to obtain a plurality of slice results, wherein the slice results include the number of slices obtained by slicing the kernel of the first SpMM operator and the dimensions of the slices; the plurality of slice results are screened using constraints, and at least one slice result that is adapted to the constraints is screened out, wherein the constraints are related to the characteristics of the device running the first SpMM operator.

[0024] In one embodiment, the processing unit is further used to filter out the preset number of slicing results from the at least one slicing result based on the device performance of the first SpMM operator running the slicing result when the number of the at least one slicing result is greater than the preset number.

[0025] In one implementation, the dimensions of different slices in the slicing results are all the same, and the dimension of the slice refers to the number of rows and columns of the matrix corresponding to the slice.

[0026] In one embodiment, the processing unit is further configured to add zero elements to slices having missing elements in the slicing result when an integer multiple of a dimension of the slices in the slicing result is different from a dimension of the first SpMM operator.

[0027] In one embodiment, the constraint conditions include register resource constraints, and the register resource constraints are related to the maximum memory of the register accessed when the device runs the slice; the processing unit is specifically used to determine the memory occupied by the resources required for the device to run the slice based on the dimension of the slice in the slice result; if the memory occupied by the resources required for the device to run the slice exceeds the maximum memory of the register accessed when the device runs the slice, the slice result corresponding to the slice is discarded.

[0028] In one embodiment, the constraint condition also includes a hardware utilization constraint condition, which is related to the number of cores of the device running the first SpMM operator; the processing unit is specifically used to discard the slicing result when the number of slices of the slicing result is less than the number of cores of the device running the first SpMM operator.

[0029] In one embodiment, the constraint condition also includes a load balancing constraint condition, which is related to a slice balance threshold, and the slice balance threshold refers to the difference between the maximum proportion of non-zero elements in the slice in the slicing result and the minimum proportion of non-zero elements in the slice in the slicing result; the processing unit is specifically used to discard the slicing result when the difference between the proportion of non-zero elements in the first slice and the proportion of non-zero elements in the second slice in the slicing result is greater than the slice balance threshold.

[0030] In one embodiment, the processing unit is specifically used to reorder the elements in each row of the sparse matrix based on the number of non-zero elements in each row of the sparse matrix calculated by the first SpMM operator; and sequentially allocate the elements of the output matrix calculated from the reordered row of elements to the slice with the least non-zero elements.

[0031] In one embodiment, the processing unit is also used to compile the filtered slicing results to obtain the code corresponding to the slicing results. When receiving the sparse matrix and the dense matrix, the code divides the sparse matrix and the dense matrix, and performs matrix multiplication to obtain the output matrix.

[0032] In a third aspect, an embodiment of the present application provides a computing device, comprising: at least one memory, and at least one processor, wherein the at least one processor is used to execute instructions stored in the at least one memory, so that the computing device can be implemented as each of the possible embodiments of the first aspect.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute various possible implementation embodiments of the first aspect.

[0034] In a fifth aspect, an embodiment of the present application provides a computer program product, characterized in that the computer program product stores instructions, and when the instructions are executed by a computer, the computer implements each possible implementation embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0036] Figure 1 A schematic diagram of a matrix multiplication method in the related art;

[0037] Figure 2 A schematic diagram of a process of optimizing a SpMM operator by an optimization system provided in an embodiment of the present application;

[0038] FIG3( a ) is a schematic diagram of segmenting a sparse matrix and a dense matrix corresponding to a slice provided in an embodiment of the present application;

[0039] FIG3( b ) is a schematic diagram of dividing a sparse matrix and a dense matrix corresponding to another slice provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of a screening process of the optimization module provided in an embodiment of the present application screening multiple slice results based on constraint conditions;

[0041] Figure 5 A schematic diagram of a flow chart of an optimization method for a SpMM operator provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of the architecture of a computing device provided in an embodiment of the present application;

[0043] Figure 7 A schematic diagram of the structure of an optimization system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0045] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0046] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0047] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0048] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0049] Matrix multiplication is a binary calculation that obtains a third matrix based on two matrices. Matrix multiplication can be applied to neural networks as a basic calculation of neural networks. Figure 1 As shown, matrix C can be obtained by multiplying matrix A and matrix B. Matrix A is an M×K matrix, where M and K are both positive integers greater than or equal to 1. M represents the number of rows of matrix A, and K represents the number of columns of matrix A. Matrix B is a K×N matrix, where N is a positive integer greater than or equal to 1. K represents the number of rows of matrix B, and N represents the number of columns of matrix B. The number of columns of matrix A is the same as the number of rows of matrix B. After matrix A is multiplied by matrix B, matrix C is obtained. Matrix C is an M×N matrix.

[0050] Generally speaking, a neural network generally includes an input layer, a fully connected layer, a convolutional layer, and an output layer. Among them, the input layer receives the original input data and converts the input data into a format that the network can process. Each neuron in the fully connected layer is connected to each neuron in the previous layer to form a fully connected network, and linear transformation and nonlinear activation are performed. After the input data passes through the fully connected layer, feature extraction and transformation can be performed layer by layer, thereby achieving effective representation learning. The convolutional layer extracts local features of the input data through convolution operations and maintains the dimension and position relationship in space. The convolutional layer allows the neural network to learn more abstract and representational features from the original pixel data. The output layer receives the input of the previous layer and converts the input result into the final output result of the network.

[0051] When the fully connected layer of a neural network performs a linear transformation, it is necessary to perform a matrix multiplication operation on the input data and the weight matrix to achieve feature extraction and conversion of the input data. The input data generally exists in the form of a matrix. When the input sequence undergoes a linear transformation at a certain layer, a new feature matrix is ​​obtained, which is the input matrix. Each element of the input matrix represents the output result of each neuron in the neural network. The weight parameter is the weight connecting different neurons in the neural network, which is used to transmit information and calculate output. In a neural network, the weight parameter exists in the form of a matrix. The weight matrix is ​​usually a network structure composed of multiple layers of self-attention layers and fully connected layers, which is used to encode and decode the input sequence.

[0052] Typically, the input matrix is ​​extremely dense and the weight matrix is ​​extremely sparse. Therefore, the input matrix can be a dense matrix and the weight matrix can be a sparse matrix. The fully connected layer of the neural network can use the sparse-dense matrix multiplication (SpMM) operator to perform matrix multiplication on the input matrix and the weight matrix.

[0053] The SpMM operator is a common matrix multiplication operation. When the SpMM operator is running, a sparse matrix can be multiplied with a dense matrix to obtain an output matrix, which is generally a dense matrix. Among them, the input sparse matrix includes at least one dimension where most of the elements are zero. All elements of the input dense matrix are non-zero. The following describes the process of the SpMM operator running matrix multiplication, defining the sparse matrix as matrix X, the dimension of matrix X is R×S, and the dense matrix as matrix Y, the dimension of matrix Y is S×T. The dense matrix calculated by the SpMM operator is matrix Z, and the dimension of matrix Z is R×T.

[0054] An operator refers to an operation that performs a mathematical calculation or transformation on input data. Operators are basic components for building neural networks and are used for data processing, feature extraction, and model training. Neural networks generally include multiple SpMM operators of different dimensions. The dimension of the SpMM operator refers to the number of rows and columns of the output matrix. SpMM operators of different dimensions can calculate dense matrices Z of different dimensions. For example, a neural network can input a sparse matrix X with R rows and a dense matrix Y with T columns into a SpMM operator with a dimension of R×T. The SpMM operator performs matrix multiplication on the sparse matrix X and the dense matrix Y to obtain a dense matrix Z with a dimension of R×T.

[0055] Before the SpMM operator performs matrix multiplication, it is necessary to modify the representation of the sparse matrix X. Exemplarily, the sparse matrix X is represented in a compressed sparse row (CSR) format. The CSR format includes a value array, a column index array, and a row offset array. Among them, the value array stores the values ​​of non-zero elements. The column index array stores the column index where the non-zero elements are located. The row offset array stores the starting position of each row of non-zero elements in the value array and column index array.

[0056] The process of matrix multiplication by the SpMM operator can be divided into two steps:

[0057] In the first step, the SpMM operator determines the range of the non-zero elements in each row of the sparse matrix X in the value array and column index array through the row offset array. Then, the SpMM operator obtains the value and column index of the non-zero elements in each row of the sparse matrix X according to the range of the value array and column index array. The values ​​and column indexes obtained by the SpMM operator can be used for subsequent matrix multiplication calculations.

[0058] In the second step, the SpMM operator multiplies the value of each row of the sparse matrix X with the dense matrix Y according to the column index of the non-zero elements in each row of the sparse matrix X to obtain the dense matrix Z. Specifically, the SpMM operator can multiply the value of the non-zero element in the i-th row of the sparse matrix X with the element in the corresponding column of the dense matrix Y element by element, and accumulate the products to obtain the element at the corresponding position of the dense matrix Z, that is, Z[i][j]=sum(X[i][k]×Y[k][j]). i, k and j are all positive integers greater than 0, and i is less than or equal to R, k is less than or equal to S, and j is less than or equal to T. By analogy, the SpMM operator can calculate each element of the dense matrix Z.

[0059] Since the dimension of the dense matrix Y may be relatively large, the dense matrix Z calculated by the SpMM operator is still dense and will occupy more memory. In order to reduce the memory occupied by the SpMM operator, the neural network can use various optimization methods to improve the performance of the SpMM operator to reduce the memory occupied by the SpMM operator.

[0060] In the related art, the neural network can optimize the SpMM operator by using optimization methods such as blocking and sparse matrix rearrangement. For example, the blocking method refers to dividing the sparse matrix into multiple blocks so that the device can process each block in parallel. Although the blocking method allows the device to process multiple blocks in parallel to improve processing efficiency, the computational load of different blocks may be unbalanced. For another example, the sparse matrix rearrangement method refers to reordering the rows of the sparse matrix and then dividing it into multiple blocks. Although the sparse matrix rearrangement method can solve the problem of possible imbalance in the computational load of different blocks, the access efficiency of the reordered sparse matrix is ​​relatively low, which increases the occupied memory.

[0061] In order to solve the defects of the optimization methods in the related art, the embodiments of the present application provide an optimization method, an optimization system, a computing device and a computing device cluster for a SpMM operator. The optimization method is generally executed by the optimization system. After receiving the SpMM operator, the optimization system can use a variety of segmentation methods to segment the kernel of the SpMM operator to obtain a variety of segmentation results. The optimization system then uses the constraints related to the characteristics of the device running the SpMM operator to screen the slicing results, and screens out several segmentation results that are suitable for the constraints, so that the SpMM operator runs according to the screened segmentation results during the reasoning process, thereby improving the computing performance of the SpMM operator. In addition, the number of slicing results screened by the optimization system using the constraints may be relatively large, resulting in a relatively large search overhead when the SpMM operator is running. The optimization system can select the concentrated slicing results with the highest performance based on the performance of the device running the slicing results, so that the SpMM operator runs according to the screened segmentation results during the reasoning process, which can further improve the computing performance of the SpMM operator.

[0062] Figure 2 Schematic diagram of the process of optimizing the SpMM operator by the optimization system provided in the embodiment of the present application. Figure 2 As shown, the optimization system 200 can be divided into a block module 210, an optimization module 220 and a generation module 230 according to the execution function.

[0063] The blocking module 210 can slice the kernel of the SpMM operator in a variety of ways to obtain a variety of slicing results. Each slicing result includes the number of slices obtained by slicing the kernel of the SpMM operator and the dimension of the slice. Each slice can calculate a sparse submatrix with a dimension of R1×S and a dense submatrix with a dimension of S×T1 to generate an output submatrix with a dimension of R1×T1. The dimension of the slice refers to the combination of R1 and T1. Among them, the kernel of the operator refers to the matrix or template of the convolution operation. The kernel of the SpMM operator is usually a two-dimensional matrix, that is, the output matrix.

[0064] In an embodiment of the present application, the partitioning module 210 can use hierarchical 2D partitioning to partition the kernel of the SpMM operator. Hierarchical 2D partitioning is a common data segmentation method, which is used to divide and organize a two-dimensional data set into multiple levels. Hierarchical two-dimensional partitioning is generally used in the fields of images, videos, spatial data, etc. The main advantage of hierarchical two-dimensional partitioning is that through hierarchical organization, specific data blocks can be selectively processed or accessed as needed without operating the entire data. Hierarchical two-dimensional partitioning is very useful in large-scale data processing and parallel computing tasks, and can improve computing efficiency and computing performance.

[0065] The block module 210 can divide the kernel of the SpMM operator into multiple levels using hierarchical two-dimensional blocks. The levels divided by the block module 210 are related to the hardware of the operation optimization system 200. The hardware of the operation optimization system 200 can be a central processing unit (CPU), a graphics processing unit (GPU), a device communication unit (DCU) or other devices with processing functions. The following takes the GPU as an example to introduce the solution protected by the embodiment of the present application.

[0066] GPUs generally have three levels, namely threads, warps and blocks. A thread is the smallest unit for performing computing tasks. In GPU programming, the program is divided into multiple small tasks, each of which is executed by a thread. A warp is the basic scheduling unit of a GPU and consists of a group of continuous threads. In CUDA programming, the size of a warp is 32, i.e., there are 32 threads in each warp. The threads in a warp execute the same instructions in parallel. A thread block is a collection of a group of threads, including multiple warps. Threads in a thread block can share local memory and perform tasks collaboratively. In an embodiment of the present application, a thread block of a GPU can process a slice, i.e., an output submatrix with a dimension of Rn×Tn. Each warp of a GPU can process an output submatrix with a dimension of Rn×Swarp. Each thread of a GPU can process an output submatrix with a dimension of Rn×1.

[0067] During the segmentation process, the segmentation module 210 can use hierarchical two-dimensional segmentation to segment the kernel of the SpMM operator to obtain multiple slices, and let each thread block of the GPU process one slice. Then, the segmentation module 210 segments the sparse matrix in the row direction to obtain multiple segmentation Atiles. Each segmentation Atile includes one or more rows of sparse matrices. The multiple rows of sparse matrices included in each segmentation Atile are sparse matrices of consecutive rows, or sparse matrices of discontinuous rows. The segmentation module 210 segments the dense matrix in the column direction to obtain multiple segmentation Btiles. Each segmentation Btile includes a dense matrix of multiple columns. The multiple columns of dense matrices included in each segmentation Btile are dense matrices of consecutive columns.

[0068] In an embodiment of the present application, a thread of a GPU can process a submatrix with R2 rows and 1 column. R2 refers to the number of rows of a sparse matrix contained in a block Atile. A thread bundle of a GPU can process a submatrix with R2 rows and T2 columns. T2 refers to the number of threads contained in the thread bundle. A thread block of a GPU can process a submatrix with R2 rows and T3 columns. T3 refers to the number of threads contained in the thread block. Exemplarily, a thread bundle of a GPU can process a block Btile with the number of columns as the thread bundle. Each thread in the thread bundle can process a dense matrix of one column in the block Btile. Among them, the output of Atile×Btile is a submatrix Cblock with a dimension of Rn×Tn.

[0069] As shown in (a) of FIG3 , when the block module 210 uses hierarchical two-dimensional blocking to slice a slice of dimension (R=4, T=2), two output sub-matrices of dimension (R=4, S=8) are sliced ​​out in the sparse matrix, and four output sub-matrices of dimension (S=8, R=2) are sliced ​​out in the dense matrix. Each thread of the GPU can process an output sub-matrix Zthread of dimension (R=4, S=1). Assume that the thread bundle of the GPU includes 2 threads, and the thread bundle can process an output sub-matrix Zwarp of dimension (R=4, S=2). The thread block of the GPU includes 1 thread bundle, and the thread block can process an output sub-matrix Zblock of dimension (R=4, T=2).

[0070] As shown in FIG3(b), when the block module 210 uses hierarchical two-dimensional blocking to slice a slice with a dimension of (R=4, T=4), two output sub-matrices with dimensions of (R=4, S=8) are sliced ​​in the sparse matrix, and two output sub-matrices with dimensions of (S=8, R=4) are sliced ​​in the dense matrix. Each thread of the GPU can process an output sub-matrix Zthread with a dimension of (R=4, S=1). Assume that the thread bundle of the GPU includes 2 threads, and the thread bundle can process an output sub-matrix Zwarp with a dimension of (R=4, S=2). The thread block of the GPU includes 2 thread bundles, and the thread block can process an output sub-matrix Zblock with a dimension of (R=4, T=4).

[0071] In the embodiment of the present application, when the block module 210 uses hierarchical two-dimensional block multi-level division, the memory access in a thread bundle of the GPU can be merged into a single memory access, which can improve the memory access efficiency of the SpMM operator. The elements processed by the thread can reuse the memory access of the dense matrix and increase the amount of calculation between two memory accesses to hide the memory access time. The Atile blocks processed by the threads in the thread block of the GPU are the same, so that the load rate of running each thread is the same. Different slices have the same or nearly the same number of rows, so that the thread blocks have similar actual register usage, which can reduce the waste of registers.

[0072] The dimensions of multiple slices of each slicing result divided by the block module 210 are the same, which can improve the load balancing of each slice. Exemplarily, when the block module 210 divides the SpMM operator of dimension 8×8 into 9 slices, it can be divided into four slices of dimension 3×3, two slices of dimension 2×3, two slices of dimension 3×2 and one slice of dimension 2×2. In order to divide the kernel of the SpMM operator into multiple slices of the same dimension, the block module 210 supplements zero elements to the slices of dimension 2×3, the slices of dimension 3×2 and the slices of dimension 2×2, and fills the slices of dimension 2×3, the slices of dimension 3×2 and the slices of dimension 2×2 into slices of dimension 3×3, so as to realize that the kernel of the SpMM operator is divided into nine slices of dimension 3×3 of the same dimension. In other embodiments, the dimensions of multiple slices of each slicing result divided by the block module 210 may be different.

[0073] The slices of the multiple slice results divided by the block module 210 include high-performance slices and low-performance slices. The optimization module 220 can use the constraint conditions to filter the slice results of the low-performance slices and retain the slice results of the high-performance slices. The constraint conditions can be register resource constraints, hardware utilization constraints, and load balancing constraints.

[0074] Taking GPU as an example, when the number of variables calculated by a thread exceeds the maximum number of registers configured by itself, the excess variables in the thread can be stored in local memory. Registers are fast memories located inside the GPU. The overhead of a thread accessing local memory is basically the same as the overhead of accessing global memory, which is much higher than the overhead of accessing registers. Since the performance of the SpMM operator is heavily dependent on memory, if the memory accessed by a thread warp running a slice of the SpMM operator exceeds the memory of the registers configured by itself, the thread warp will access the local memory, resulting in reduced performance of the slice.

[0075] Therefore, the register resource constraint is related to the maximum memory of the register accessed when the warp runs the slice. When the memory of the register accessed when the warp runs the slice exceeds the maximum memory of the register, the optimization module 220 may discard the slice result corresponding to the slice and retain the slice results corresponding to the slices in other cases.

[0076] In one embodiment, the optimization system 200 can use a data embedding method to obtain the number of variables for each thread and detect the usage of registers. When the kernel of a SpMM operator's demand for registers exceeds the limit of the thread block, the kernel of the SpMM operator cannot fully utilize the parallel computing capabilities of the GPU. The number of registers required for a thread block can be calculated by multiplying the use of registers in the thread by the number of threads. Each thread in each thread block needs to use a certain number of registers to store variables, intermediate results, and other data required for calculation. The optimization module 220 can compare the number of registers required for the thread block with the capacity of the register, and delete the slice results corresponding to the slices that exceed the capacity of the register.

[0077] Figure 4 Schematic diagram of the screening process of the optimization module provided in the embodiment of the present application for screening multiple slice results based on constraint conditions. Figure 4 As shown, the two-dimensional coordinates of each cell represent the dimensions of the candidate two-dimensional slices. Dark gray, gray, and light gray represent slices removed by register resource constraints, hardware utilization constraints, and load balancing constraints, respectively. The size of the search space is reduced from 8×8=64 to 6 (white cells). Assume that the maximum number of registers for a thread is 6 and the maximum number of registers in a thread block is 24. After the optimization module 220 uses the register resource constraints for screening, the maximum number of registers of the slice is reduced from 64 to 38.

[0078] The GPU can divide a thread block into multiple thread bundles and make each thread bundle a single instruction, multiple thread (SIMT) execution unit. When the number of threads in a thread block is not an integer multiple of the number of threads in a thread bundle, the number of threads in one thread bundle is less than the number of threads in other thread bundles, resulting in a relatively low utilization rate of one thread bundle. In the embodiment of the present application, when a slice can process an output submatrix of a dimension of Rn×Tn, the number of threads of the thread block corresponding to the slice is Tn. The optimization module 220 can use Tn as a multiple of the thread bundle size to limit the search space.

[0079] In one embodiment, the GPU has multiple stream processors for executing parallel computing tasks, so that the GPU can efficiently perform large-scale parallel computing. The optimization module 220 can assign each thread block to a stream processor and let the stream processor process the thread block. If the number of thread blocks is less than the number of stream processors of the GPU, some stream processors will be idle, resulting in the inability to fully utilize the hardware resources of the GPU.

[0080] Therefore, the hardware utilization constraint is related to the number of stream processors running the SpMM operator. The optimization module 220 may discard the slicing result when the number of slices of the slicing result is less than the number of stream processors running the SpMM operator, and retain the slicing results corresponding to the slices in other cases.

[0081] like Figure 4 As shown, it is assumed that the size of the thread warp is 2 and the number of stream processors is 8. After the optimization module 220 filters the hardware utilization constraint, the search space is reduced from 38 to 11.

[0082] If the dimensions of the slices divided by the block module 210 are not appropriate, the load of running different slice thread bundles is unbalanced, which will reduce the computing performance of the SpMM operator. For example, when the dimension of the SpMM operator is 8×8, the block module 210 can divide a slice with a dimension of 7×7, a slice with a dimension of 1×7, a slice with a dimension of 7×1, and a slice with a dimension of 1×1. At this time, the load of the slice with a dimension of 7×7 is much different from that of other slices, resulting in extremely unbalanced load rates of multiple slices of the slicing result.

[0083] In an embodiment of the present application, the optimization module 220 may need to find a slice dimension with the same number of slices and a more balanced slice load, and delete slices that do not meet the load balancing condition, which can improve the performance of the kernel of the SpMM operator. When the dimensions of multiple slices in the slicing result are the same, the optimization module 220 can detect the proportion of non-zero elements of each slice in all elements of the entire slice. The higher the proportion of non-zero elements in the slice, the fewer zero elements filled in the slice, indicating that the real dimensions of the slices are closer to the same. The real dimensions of multiple slices in the slicing result are the same, and the load rate of the slicing result is more balanced.

[0084] Therefore, the load balancing constraint condition is related to the balance threshold of the slice. The balance threshold of the slice refers to the difference between the maximum proportion of non-zero elements in the slice in the slicing result and the minimum proportion of non-zero elements in the slice in the slicing result. The optimization module 220 can discard the slicing result when the difference between the maximum proportion of non-zero elements in the slice in the slicing result and the minimum proportion of non-zero elements in the slice is greater than the balance threshold of the slice, and retain the slicing results corresponding to the slices in other cases.

[0085] like Figure 4 As shown, the dimension of the output matrix Z is Rn×Tn, and let Rn=6. Let the sparse matrix X be divided into two blocks Atile. The first block Atile has 6 rows and 24 non-zero elements. The second block Atile has only 2 rows and 8 non-zero elements. At this time, the load of the two slices is unbalanced. The GPU can set Rn of the output matrix Z to 4 so that each block Atile has 4 rows. The optimization module 220 keeps the number of slices of the SpMM operator the same, so that the load of the slices corresponding to different block Atiles is more balanced.

[0086] Similarly, let Tn=6. The first tile Btile has 6 columns. The second tile Btile has only 2 columns. At this point, the loads of the two slices are unbalanced. The optimization module 220 can set Tn of the output matrix Z to 4, so that each tile Btile has 4 columns. The optimization module 220 keeps the number of slices of the SpMM operator the same, so that the loads of the slices corresponding to different tiles Atile are more balanced.

[0087] The optimization module 220 can measure the load balance of the slice corresponding to the tile Atile by calculating the coefficient of variation of the number of non-zero elements in the tile Atile, and measure the load balance of the slice corresponding to the tile Btile by the filling ratio of the column in the dense matrix. The optimization module 220 can use COVrow to set the threshold Throw, and use COLwaste to set the threshold THcol. Assuming that the threshold THrow and the threshold THcol are both set to 0.25, Figure 4 The search space in is reduced from 11 to 6 after the load balancing constraint.

[0088] Since there are a large number of zero elements in the sparse matrix, matrix multiplication of the zero elements in the sparse matrix with the elements of the dense matrix may produce zero elements, which will affect the number of zero elements in the slice. In the embodiment of the present application, before the optimization module 220 uses the load balancing constraint to filter the low-performance slices, it is necessary to reorder the sparse matrix. The optimization module 220 changes the order of each block Atile of the sparse matrix, and reorders each block Atile according to the number of non-zero elements. It can reduce the maximum number of memory accesses in all slices while maintaining the computational load balance between all slices.

[0089] The optimization module 220 can use an approximate algorithm to first sort the elements of each row of the sparse matrix according to the number of non-zero elements. Then, the optimization module 220 sequentially distributes the elements of each row to the slice with the least non-zero elements according to the order of arrangement, so that the non-zero elements between different slices are kept close, reducing the impact of zero elements in the sparse matrix on the load balancing constraint condition, thereby achieving balanced distribution of computing load.

[0090] For example, assuming that the dimension of the slice is Rn×Tn, the maximum number of memory accesses of the tile Atile is:

[0091]

[0092] Among them, Astile is a group of block Atiles, and the dimension of the block Atile is restricted by R1 so that the register usage of all threads is close.

[0093] The optimization module 220 can use formula (1) to calculate the number of non-zero elements in each row and the entire sparse matrix, which are expressed as nnzsrow and nnzmatrix (rows 3-4) respectively. Then, the optimization module 220 sorts the rows of the sparse matrix according to nnzrow and deletes the empty rows (rows 5-6). The rows of the sparse matrix are processed in increasing order of nnzrow. For each row of the sparse matrix, the optimization module 220 can merge each row with each block Atile and sort the temp Astiles according to the nnc in the block Atile (rows 11-12). The optimization module 220 can start from the block Atile with the minimum nnc and the dimension of the block Atile not exceeding the dimension limit of the block Atile. If the nnz in the block Atile does not exceed the nnz limit (THnnz), this row is assigned to the block Atile. On the contrary, the optimization module 220 selects the next block Atile for judgment. If the nnz of all unfilled tiles is greater than the limit, this row is assigned to the tile with the smallest nnc (lines 13-23). ​​During the reordering process, the optimization module 220 limits the number of non-zero elements in each tile so that the nnz between different tiles is close, thereby achieving a balance in the computational load.

[0094] The number of slice results screened out by the optimization module 220 using the constraint conditions may be relatively large, resulting in a relatively large search overhead for the optimization system 200. The optimization module 220 may construct a sorting model based on factors such as load balance, inference latency, and memory usage. The optimization module 220 inputs the screened slice results into the sorting model, and may output a set number of slice results, further improving the computing performance of the SpMM operator.

[0095] The sorting model can be implemented based on the random forest algorithm. The random forest algorithm is an integrated learning method that combines multiple decision trees to perform classification and regression tasks. The main idea of ​​the random forest algorithm is to form a "forest" by constructing multiple decision trees, and use random sampling and feature random selection methods to enhance the diversity of each decision tree. In the embodiment of the present application, when constructing the sorting model, the optimization module 220 can set factors such as load balance, reasoning delay, and occupied memory as decision trees. Multiple slice results are input into the sorting model, and the sorting model can output a set number of slice results based on the principles of highest load balance, shortest reasoning delay, and minimum occupied memory.

[0096] After receiving the several slicing results filtered by the optimization module 220, the generation module 230 can respectively perform actual code generation, code compilation and code execution on the slicing results to obtain the running code of each slice. The generation module 230 can store the running code in the memory. After receiving the sparse matrix and the dense matrix, the neural network can select the SpMM operator of the corresponding dimension based on the dimension of the output matrix calculated by the sparse matrix and the dense matrix, and select the optimal slicing result of the SpMM operator. The SpMM operator can retrieve the running code corresponding to the slicing result from the memory, implement matrix multiplication of the sparse matrix and the dense matrix, and calculate the output matrix.

[0097] In an embodiment of the present application, after receiving the SpMM operator, the optimization system 200 can use a variety of segmentation methods to segment the kernel of the SpMM operator to obtain a variety of segmentation results. The optimization system 200 can screen various slices of the SpMM operator based on a search algorithm that combines constraints and sorting models to obtain high-performance slices of the SpMM operator. In addition, the optimization system 200 uses a sparse matrix rearrangement algorithm to reduce the number of memory accesses of each slice while considering the load balancing between slices, thereby improving the performance of the generated SpMM operator.

[0098] The above is an introduction to the optimization system provided in the embodiment of the present application. Next, based on the above content, the optimization method provided in the embodiment of the present application is introduced.

[0099] Figure 5 FIG. 2 is a flow chart showing a method for optimizing a SpMM operator provided in an embodiment of the present application. The method can be executed by an optimization system 200. Figure 5 As shown, the optimization method may include the following steps:

[0100] S501, the optimization system 200 obtains the SpMM operator from the neural network.

[0101] The SpMM operator is an operator that calculates sparse matrix multiplication. It can be used to receive sparse matrices and dense matrices, and perform matrix multiplication on the sparse matrix and the dense matrix to obtain an output matrix. Different SpMM operators can calculate output matrices of different dimensions. For example, a SpMM operator with a dimension of (R, T) can perform matrix multiplication on a sparse matrix with a dimension of (R, S) and a dense matrix with a dimension of (S, T) to obtain an output matrix with a dimension of (R, T).

[0102] S502, the optimization system 200 divides the kernel of the SpMM operator by a plurality of dividing methods to obtain a plurality of dividing results.

[0103] When the optimization system 200 divides the kernel of the SpMM operator by multiple division methods, the kernel of the SpMM operator can be divided into multiple slices of the same dimension, or the kernel of the SpMM operator can be divided into multiple slices of different dimensions. Assume that when the optimization system 200 divides the SpMM operator of dimension 8×8 into 9 slices, it can be divided into four slices of dimension 3×3, two slices of dimension 2×3, two slices of dimension 3×2 and one slice of dimension 2×2. In order to divide the kernel of the SpMM operator into multiple slices of the same dimension, the optimization system 200 adds zero elements to the slices of dimension 2×3, the slices of dimension 3×2 and the slices of dimension 2×2, and fills the slices of dimension 2×3, the slices of dimension 3×2 and the slices of dimension 2×2 into slices of dimension 3×3, so as to realize that the kernel of the SpMM operator is divided into nine slices of dimension 3×3 of the same dimension.

[0104] Taking the hardware running the optimization system 200 as a GPU as an example, the GPU can be divided into three levels, namely threads, thread bundles and thread blocks. A thread is the smallest task that performs a computing task. A thread bundle includes multiple threads, and multiple threads are run in parallel to perform computing tasks. A thread block includes multiple thread bundles, and multiple thread bundles are run in parallel to perform computing tasks. In an embodiment of the present application, the optimization system 200 can use hierarchical two-dimensional blocking to divide the kernel of the SpMM operator to obtain multiple slices. A thread block of the GPU can process one slice. Then, the optimization system 200 divides the sparse matrix in the direction of rows, and the sparse matrix of one or more rows after the division is used as a block Atile. The sparse matrices of multiple rows included in each block Atile are sparse matrices of several consecutive rows, or sparse matrices of several discontinuous rows. Hierarchical two-dimensional blocking divides the dense matrix in the direction of columns, and the dense matrices of multiple columns after the division are used as block Btile. The dense matrices of multiple columns included in each block Btile are dense matrices of several consecutive columns. A GPU thread warp can process a tile Btile with the same number of columns as the thread warp. Each thread in the thread warp can process a dense matrix of one column in the tile Btile. The output of Atile×Btile is a submatrix Cblock with the dimension Rn×Tn.

[0105] In the embodiment of the present application, when the optimization system 200 utilizes hierarchical two-dimensional block multi-level division, the memory access of multiple threads in a thread bundle of the GPU can be merged into a single memory access, which can improve the memory access efficiency of the SpMM operator. The elements processed by the threads can reuse the memory access of the dense matrix and increase the amount of calculation between two memory accesses to hide the memory access time. The output sub-matrices processed by the threads in the thread block of the GPU are the same, so that the load rate of each thread is the same. Different slices have the same or nearly the same number of rows, so that each thread block uses the same memory of the register, which can reduce the waste of registers.

[0106] S503, the optimization system 200 uses the constraint conditions to screen the multiple slicing results, and screens out at least one slicing result that meets the constraint conditions.

[0107] The optimization system 200 may filter the slicing results of low-performance slices using constraints and retain the slicing results of high-performance slices. The constraints may be register resource constraints, hardware utilization constraints, and load balancing constraints.

[0108] The register resource constraint is related to the maximum memory of the register accessed when the thread bundle runs the slice. In the embodiment of the present application, when the number of variables calculated by the thread exceeds the maximum number of registers configured by itself, the excess variables in the thread can be stored in the local memory. The overhead of the thread accessing the local memory is basically the same as the overhead of accessing the global memory, which is much higher than the overhead of accessing its own registers. The optimization system 200 can discard the slice result corresponding to the slice when the memory accessed to the register exceeds the maximum memory of the register when the thread bundle runs the slice, and retain the slice results corresponding to the slices in other cases.

[0109] The hardware utilization constraint is related to the number of stream processors running the SpMM operator. In an embodiment of the present application, there are multiple stream processors inside the GPU, which are responsible for executing parallel computing tasks, so that the GPU can efficiently perform large-scale parallel computing. The optimization system 200 can assign each thread block to a stream processor and let the stream processor process the thread block. If the number of thread blocks is less than the number of stream processors of the GPU, some stream processors will be idle, resulting in the inability to fully utilize the hardware resources of the GPU. The optimization module 220 of the optimization system 200 can discard the slicing result when the number of slices of the slicing result is less than the number of stream processors running the SpMM operator, and retain the slicing results corresponding to the slices in other cases.

[0110] The load balancing constraint is related to the balance threshold of the slice. The balance threshold of the slice refers to the difference between the maximum value of the proportion of non-zero elements in the slice in the slicing result and the minimum value of the proportion of non-zero elements in the slice in the slicing result. In the embodiment of the present application, when the kernel of the SpMM operator is not properly divided into slice dimensions, many slices need to be supplemented with more zero elements. The more zero elements a slice is supplemented with, the more unbalanced the load rate of the slice with other slices. The more slices that need to be supplemented with zero elements, the more unbalanced the load rate of the kernel of the SpMM operator. The optimization system 200 optimization module 220 can discard the slice result when the difference between the maximum value of the proportion of non-zero elements in the slice in the slicing result and the minimum value of the proportion of non-zero elements in the slice is greater than the balance threshold of the slice, and retain the slice results corresponding to the slices in other cases.

[0111] Since there are a large number of zero elements in the sparse matrix, the matrix multiplication of the zero elements in the sparse matrix and the elements of the dense matrix may produce zero elements, which will affect the number of zero elements in the slice. Before the optimization system 200 uses the load balancing constraint to filter the low-performance slices, it is necessary to reorder the sparse matrix. In an embodiment of the present application, the optimization system 200 reorders the elements of each row in the sparse matrix based on the number of non-zero elements in each row in the sparse matrix calculated by the SpMM operator. Then, the optimization system 200 can sequentially distribute the elements of the output matrix calculated by a row of elements after the reordering to the slice with the least non-elements, so that the non-zero elements between different slices are kept close, reducing the impact of the zero elements in the sparse matrix on the load balancing constraint, thereby achieving balanced distribution of computing load.

[0112] S504, when the number of at least one slicing result is greater than a preset number, the optimization system 200 filters out a preset number of slicing results from at least one slicing result based on the device performance of the SpMM operator running the slicing results.

[0113] The number of slice results screened out by the optimization system 200 using the constraint conditions may be relatively large, resulting in a relatively large search overhead for the optimization system 200. The optimization system 200 can construct a sorting model based on factors such as load balance, reasoning delay, and memory occupied. The optimization module 220 inputs the screened slice results into the sorting model and can output a set number of slice results. In an embodiment of the present application, when constructing the sorting model, the optimization module 220 can use a random forest algorithm to set factors such as load balance, reasoning delay, and memory occupied as a decision tree. Multiple slice results are input into the sorting model, and the sorting model can output a set number of slice results based on the principles of highest load balance, shortest reasoning delay, and minimum memory occupied.

[0114] S505, the optimization system 200 compiles the screened slicing results to obtain codes corresponding to the slicing results.

[0115] After receiving the screened slicing results, the optimization system 200 can perform actual code generation, code compilation and code execution on the slicing results respectively to obtain the running code of each slice. The optimization system 200 can store the running code in the memory. After receiving the sparse matrix and the dense matrix, the neural network can select the SpMM operator of the corresponding dimension based on the dimension of the output matrix calculated by the sparse matrix and the dense matrix, and select the optimal slicing result of the SpMM operator. The SpMM operator can retrieve the running code corresponding to the slicing result from the memory, implement matrix multiplication of the sparse matrix and the dense matrix, and calculate the output matrix.

[0116] In an embodiment of the present application, after receiving the SpMM operator, the optimization system 200 can use a variety of slicing methods to slice the kernel of the SpMM operator to obtain a variety of slicing results. The optimization system then uses the constraints related to the characteristics of the device running the SpMM operator to filter the slicing results, and filters out several slicing results that are suitable for the constraints, so that the SpMM operator runs according to the filtered slicing results during the reasoning process, thereby improving the computing performance of the SpMM operator. In addition, the number of slicing results filtered out by the optimization system using the constraints may be relatively large, resulting in a relatively large search overhead when the SpMM operator is running. The optimization system can select the concentrated slicing result with the highest performance based on the performance of the device running the slicing results, so that the SpMM operator runs according to the filtered slicing results during the reasoning process, which can further improve the computing performance of the SpMM operator.

[0117] like Figure 6 As shown, the computing device generally includes an application, an application programming interface (API), an operating system, an instruction set architecture (ISA), a hardware system, etc. When the neural network can be deployed on the computing device through tensorflow lite (tf-lite), Tensor RT, open CV, and mobile neural network (MNN) deployment tools, the running code generated by the optimization system 200 is stored in the memory of the hardware system and deployed at the application level. When the computing device runs the neural network, the neural network can call the running code in the memory to implement the SpMM operator to perform reasoning according to the set slice results.

[0118] Figure 7 Schematic diagram of the structure of an optimization system provided in the embodiment of the present application. Figure 7 As shown, the optimization system 700 can be divided into a transceiver unit 710 and a processing unit 720 according to the execution function. The specific implementation process of the optimization system 700 is as follows:

[0119] The transceiver unit 710 is used to obtain a first SpMM operator from a neural network. The processing unit 720 is used to slice the kernel of the first SpMM operator in a plurality of slicing methods to obtain a plurality of slicing results. The slicing results include the number of slices obtained by slicing the kernel of the first SpMM operator and the dimensions of the slices. The processing unit 720 is used to filter the plurality of slicing results using constraint conditions to filter out at least one slicing result that is adapted to the constraint conditions. The constraint conditions are related to the characteristics of the device running the first SpMM operator running the first SpMM operator.

[0120] In one embodiment, the processing unit 720 is further used to filter out a preset number of slicing results from at least one slicing result based on the device performance of the first SpMM operator running the slicing results when the number of at least one slicing result is greater than a preset number.

[0121] In one embodiment, the dimensions of different slices in the slice result are the same. The dimension of the slice refers to the number of rows and columns of the matrix corresponding to the slice.

[0122] In one embodiment, the processing unit 720 is further configured to supplement zero elements in the slices having missing elements in the slice result when an integer multiple of the dimension of the slice in the slice result is different from the dimension of the first SpMM operator.

[0123] In one embodiment, the constraint condition includes a register resource constraint condition, and the register resource constraint condition is related to the maximum memory of the register accessed when the device runs the slice. The processing unit 720 is specifically used to determine the memory occupied by the resources required for the device to run the slice based on the dimension of the slice in the slice result. The processing unit 720 is specifically used to discard the slice result corresponding to the slice when the memory occupied by the resources required for the device to run the slice exceeds the maximum memory of the register accessed when the device runs the slice.

[0124] In one embodiment, the constraint condition further includes a hardware utilization constraint condition. The hardware utilization constraint condition is related to the number of cores of the device running the first SpMM operator. The processing unit 720 is specifically configured to discard the slicing result when the number of slices of the slicing result is less than the number of cores of the device running the first SpMM operator.

[0125] In one embodiment, the constraint condition also includes a load balancing constraint condition. The load balancing constraint condition is related to the balance threshold of the slice. The balance threshold of the slice refers to the difference between the maximum proportion of non-zero elements in the slice in the slicing result and the minimum proportion of non-zero elements in the slice in the slicing result. The processing unit 720 is specifically used to discard the slicing result when the difference between the proportion of non-zero elements in the first slice and the proportion of non-zero elements in the second slice in the slicing result is greater than the balance threshold of the slice.

[0126] In one embodiment, the processing unit 720 is specifically configured to reorder the elements of each row in the sparse matrix based on the number of non-zero elements in each row in the sparse matrix calculated by the first SpMM operator. The processing unit 720 is specifically configured to sequentially assign the elements of the output matrix calculated from the reordered row of elements to the slice with the least non-zero elements.

[0127] In one embodiment, the processing unit 720 is further used to compile the screened slicing results to obtain codes corresponding to the slicing results. When receiving the sparse matrix and the dense matrix, the code divides the sparse matrix and the dense matrix, and performs matrix multiplication to obtain an output matrix.

[0128] The present application also provides a computing device, which includes a processor, and the processor can execute the following Figure 2-Figure 7 And the above corresponding protected technical solutions, so that the computing device has the technical effect of the above protected technical solutions.

[0129] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer executes the above Figure 2-Figure 7 and any method described in the corresponding description.

[0130] In an embodiment of the present application, a computer program product is also provided, wherein the computer program product stores instructions, and when the instructions are executed by a computer, the computer implements the above Figure 2-Figure 7 and any method described in the corresponding description.

[0131] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0132] In addition, various aspects or features of the embodiments of the present application can be implemented as methods, devices or products using standard programming and / or engineering techniques. The term "product" used in this application covers computer programs that can be accessed from any computer-readable device, carrier or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks or tapes, etc.), optical disks (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards and flash memory devices (e.g., erasable programmable read-only memory (EPROM), cards, sticks or key drives, etc.). In addition, the various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0133] In the above embodiment, Figure 7 The optimization system 700 in can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0134] It should be understood that in various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0135] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0136] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0137] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or an access network device, etc.) to perform all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0139] The above is only a specific implementation of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited to this. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A method for optimizing a sparse-dense matrix multiplication SpMM operator, characterized in that: The method comprises: Get the first SpMM operator from the neural network; The kernel of the first SpMM operator is sliced ​​by multiple slicing methods to obtain multiple slicing results, wherein the slicing results include the number of slices obtained by slicing the kernel of the first SpMM operator and the dimensions of the slices; The plurality of slicing results are screened using a constraint condition to screen out at least one slicing result that is adapted to the constraint condition, wherein the constraint condition is related to a feature of a device running the first SpMM operator.

2. The method according to claim 1, characterized in that The method further comprises: In a case where the number of the at least one slicing result is greater than a preset number, the preset number of slicing results are screened out from the at least one slicing result based on the device performance of the first SpMM operator that runs the slicing results.

3. The method according to claim 1 or 2, characterized in that: The dimensions of different slices in the slicing results are all the same, and the dimension of the slice refers to the number of rows and columns of the matrix corresponding to the slice.

4. The method according to claim 2, characterized in that: Before screening the plurality of slicing results by using the constraint condition to screen out at least one slicing result that is suitable for the constraint condition, the method further includes: In the case where an integer multiple of a dimension of a slice in the slicing result is different from a dimension of the first SpMM operator, zero elements are added to the slices in which there are missing elements in the slicing result.

5. The method according to any one of claims 1 to 4, characterized in that: The constraint condition includes a register resource constraint condition, and the register resource constraint condition is related to a maximum memory accessed to a register when the device runs a slice; The using the constraint condition to screen the multiple slicing results to select at least one slicing result that is suitable for the constraint condition specifically includes: Determine, based on the dimensions of the slices in the slicing result, the memory occupied by the resources required by the device to run the slices; When the memory occupied by the resources required by the device to run the slice exceeds the maximum memory of the register accessed when the device runs the slice, the slice result corresponding to the slice is discarded.

6. The method according to any one of claims 1 to 5, characterized in that: The constraint condition further includes a hardware utilization constraint condition, and the hardware utilization constraint condition is related to the number of cores of the device running the first SpMM operator; The using the constraint condition to screen the multiple slicing results to select at least one slicing result that is suitable for the constraint condition specifically includes: When the number of slices of the slicing result is less than the number of cores of the device running the first SpMM operator, the slicing result is discarded.

7. The method according to any one of claims 1 to 6, characterized in that: The constraint condition further includes a load balancing constraint condition, which is related to a slice balance threshold, and the slice balance threshold refers to a difference between a maximum value of a proportion of non-zero elements in a slice in a slicing result and a minimum value of a proportion of non-zero elements in a slice in a slicing result; The using the constraint condition to screen the multiple slicing results to select at least one slicing result that is suitable for the constraint condition specifically includes: When the difference between the proportion of non-zero elements in the first slice and the proportion of non-zero elements in the second slice in the slice result is greater than the balance threshold of the slice, the slice result is discarded.

8. The method according to claim 7, characterized in that The kernel of the first SpMM operator is sliced ​​by multiple slicing methods to obtain multiple slicing results, specifically including: Reordering the elements of each row in the sparse matrix based on the number of non-zero elements in each row in the sparse matrix calculated by the first SpMM operator; The elements of the output matrix calculated by reordering a row of elements are assigned to the slice with the least non-elements in turn.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: The filtered slicing results are compiled to obtain codes corresponding to the slicing results. When receiving a sparse matrix and a dense matrix, the code segments the sparse matrix and the dense matrix, and performs matrix multiplication to obtain an output matrix.

10. An optimization system, characterized in that, include: A transceiver unit, used for obtaining a first SpMM operator from a neural network; A processing unit, configured to slice the kernel of the first SpMM operator in a plurality of slice modes to obtain a plurality of slice results, wherein the slice results include the number of slices obtained by slicing the kernel of the first SpMM operator and the dimensions of the slices; The plurality of slicing results are screened using a constraint condition to screen out at least one slicing result that is adapted to the constraint condition, wherein the constraint condition is related to a feature of a device running the first SpMM operator.

11. The optimization system according to claim 10, characterized in that: The processing unit is further configured to filter out the preset number of slicing results from the at least one slicing result based on the device performance of the first SpMM operator that runs the slicing result when the number of the at least one slicing result is greater than the preset number.

12. The optimization system according to claim 10 or 11, characterized in that: The dimensions of different slices in the slicing results are all the same, and the dimension of the slice refers to the number of rows and columns of the matrix corresponding to the slice.

13. The optimization system according to claim 12, characterized in that: The processing unit is further configured to, when an integer multiple of a dimension of a slice in the slicing result is different from a dimension of the first SpMM operator, supplement zero elements in a slice having missing elements in the slicing result.

14. The optimization system according to any one of claims 10 to 13, characterized in that: The constraint condition includes a register resource constraint condition, and the register resource constraint condition is related to a maximum memory accessed to a register when the device runs a slice; The processing unit is specifically configured to determine, based on the dimension of the slice in the slicing result, the memory occupied by the resources required by the device to run the slice; When the memory occupied by the resources required by the device to run the slice exceeds the maximum memory of the register accessed when the device runs the slice, the slice result corresponding to the slice is discarded.

15. The optimization system according to any one of claims 10 to 14, characterized in that: The constraint condition further includes a hardware utilization constraint condition, and the hardware utilization constraint condition is related to the number of cores of the device running the first SpMM operator; The processing unit is specifically configured to discard the slicing result when the number of slices of the slicing result is less than the number of cores of the device running the first SpMM operator.

16. The optimization system according to any one of claims 10 to 15, characterized in that: The constraint condition further includes a load balancing constraint condition, which is related to a slice balance threshold, and the slice balance threshold refers to a difference between a maximum value of a proportion of non-zero elements in a slice in a slicing result and a minimum value of a proportion of non-zero elements in a slice in a slicing result; The processing unit is specifically configured to discard the slicing result when a difference between a proportion of non-zero elements in a first slice and a proportion of non-zero elements in a second slice in the slicing result is greater than a balance threshold of the slice.

17. The optimization system according to claim 16, characterized in that: The processing unit is specifically configured to reorder the elements of each row in the sparse matrix based on the number of non-zero elements in each row in the sparse matrix calculated by the first SpMM operator; The elements of the output matrix calculated by reordering a row of elements are assigned to the slice with the least non-elements in turn.

18. The optimization system according to any one of claims 10 to 17, characterized in that: The processing unit is also used to compile the filtered slicing results to obtain the code corresponding to the slicing results. When receiving the sparse matrix and the dense matrix, the code divides the sparse matrix and the dense matrix, and performs matrix multiplication to obtain the output matrix.

19. A computing device, characterized in that include: at least one memory, At least one processor, the at least one processor is used to execute instructions stored in the at least one memory, so that the computing device performs the method according to any one of claims 1-9.

20. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 9.

21. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, enable the computer to implement the method according to any one of claims 1 to 9.

Images