Method and storage medium for accelerating sparse matrix calculation on tensor processing unit

Abstract

The present application discloses a method and storage medium for accelerating sparse matrix calculation on a tensor processing unit, the method comprising: obtaining a first matrix, wherein the first matrix includes row index values ​​or column index values, and the first matrix is ​​a sparse matrix obtained based on an input sequence; based on the row index values ​​or column index values, reordering the rows and columns of the first matrix to obtain a second matrix, so that the cache hit rate of the second matrix relative to the first matrix is ​​higher; reading a third matrix from the memory of a GPU, and operating the second matrix and the third matrix, wherein the third matrix refers to a matrix that operates with the second matrix, and the third matrix is ​​obtained based on the input sequence. Through this technology, and performing matrix operations on tensor cores, redundant computing operations are reduced, the cache hit rate during operation is improved, and efficient computing of sparse matrices and optimized utilization of computing resources are achieved, greatly improving the overall performance of matrix computing.

Description

Technical Field

[0001] The present application relates to the field of tensor operation acceleration, and more specifically, to a method and storage medium for accelerating sparse matrix calculations on a tensor processing unit. Background Art

[0002] With the development of technologies such as neural networks, the requirements for the efficiency of matrix calculations are becoming increasingly stronger.

[0003] There is a type of matrix called sparse matrix, which has many zero elements. Using the existing conventional matrix calculation method, processing the elements in the sparse matrix one by one will occupy a large amount of computing resources and memory resources, resulting in the consumption of computing resources and memory resources.

[0004] Accordingly, the art needs a new accelerated sparse matrix calculation solution to solve the above problems. Summary of the invention

[0005] In order to overcome the above-mentioned defects, the present invention is proposed to provide a solution or at least partially solve the technical problem in the prior art that sparse matrix calculations occupy a large amount of computing resources and memory resources.

[0006] In a first aspect, the present invention provides a method for accelerating sparse matrix calculations on a tensor processing unit, the method comprising: obtaining a first matrix, wherein the first matrix includes row index values ​​or column index values, and the first matrix is ​​a sparse matrix obtained based on an input sequence; based on the row index values ​​or column index values, reordering the rows and columns of the first matrix to obtain a second matrix, so that the cache hit rate of the second matrix relative to the first matrix is ​​higher; reading a third matrix from the memory of a GPU, and performing operations on the second matrix and the third matrix, wherein the third matrix refers to a matrix that operates with the second matrix, and the third matrix is ​​obtained based on the input sequence.

[0007] In a second aspect, a control device is provided, which includes a processor and a storage device, wherein the storage device is suitable for storing multiple computer programs, and the computer programs are suitable for being loaded and run by the processor to execute the method for accelerating sparse matrix calculations on a tensor processing unit described in any of the technical solutions of the method for accelerating sparse matrix calculations on the tensor processing unit.

[0008] In a third aspect, a computer-readable storage medium is provided, which stores a plurality of computer programs, wherein the computer programs are suitable for being loaded and run by a processor to execute the method for accelerating sparse matrix calculations on a tensor processing unit described in any one of the technical solutions of the method for accelerating sparse matrix calculations on the tensor processing unit.

[0009] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects:

[0010] In the technical solution of the present invention, by reordering the sparse matrix, the technology effectively improves the concentration of non-zero elements and improves the cache hit rate during operation. After that, the double buffering technology is used to load the compressed matrix and the third matrix, so that the data loading and calculation process can be carried out in parallel, making full use of the computing power of the tensor processing unit and significantly improving the computing efficiency. In general, the technology reduces redundant computing operations, realizes efficient computing of sparse matrices and optimized utilization of computing resources, and greatly improves the overall performance of matrix calculations.

[0011] Furthermore, by adjusting the task allocation of thread blocks in a load balancing manner, the maximum utilization of computing resources is achieved, the waiting time between thread blocks is reduced, and the overall performance and efficiency of the tensor processing unit when performing sparse matrix calculations are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0013] Figure 1 It is a flowchart of main steps of a method for accelerating sparse matrix calculation on a tensor processing unit according to one embodiment of the present invention;

[0014] Figure 2a is a first matrix schematic diagram of a method for accelerating sparse matrix calculation on a tensor processing unit according to an embodiment of the present invention;

[0015] Figure 2b It is a schematic diagram of the graph structure obtained according to Figure a;

[0016] Figure 2c It is a tree diagram after merging and expanding Figure a;

[0017] Figure 2d It is a schematic diagram of the corresponding graph structure obtained from a third array obtained according to Figure c;

[0018] Figure 2e yes Figure 2d The corresponding second matrix schematic diagram;

[0019] Figure 3 is a schematic diagram of obtaining a second matrix by accelerating sparse matrix calculation on a tensor processing unit according to an embodiment of the present invention;

[0020] Figure 4 is a schematic diagram of compressing a matrix according to an embodiment of the present invention;

[0021] Figure 5 is a schematic diagram of reading and operating a matrix according to an embodiment of the present invention;

[0022] Figure 6 is a timing diagram of loading and operating a sparse matrix according to an embodiment of the present invention;

[0023] Figure 7 FIG. 4 is a schematic diagram of a thread block processing a matrix block according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] Here we first explain some terms involved in the present invention.

[0027] Sparse matrix: A sparse matrix is ​​a matrix where most of the elements are zero. Sparse matrices are commonly used in graph neural networks and scientific computing.

[0028] Dense Matrix: A dense matrix is ​​a matrix in which most of the elements are not zero.

[0029] Dense matrix operations: Dense matrix operations refer to mathematical operations performed on matrices (either sparse or dense), such as matrix multiplication, addition, or inverse matrix calculations. In dense matrix operations, each element is involved in the calculation, and the time taken to perform the operation is only related to the number of elements in the matrix involved in the calculation. Therefore, dense matrix operations require more computing power and also require more storage resources and computing time when using dense matrix operations.

[0030] With the rapid development of technologies such as neural networks and deep learning, the demand for computing power is also rising rapidly. Among the many types of computing power requirements, matrix calculations account for a large part.

[0031] As the demand for matrix calculations increases, many manufacturers have launched various tensor processing units that specialize in matrix calculations. Tensor processing units are used to perform large numbers of parallel matrix multiplications and other matrix operations, so they are very suitable for neural network training and reasoning.

[0032] Common existing tensor processing units include Tensor Core (tensor computing core), TPU (Tensor Processing Unit, tensor operation core), etc. Among them, Tensor Core is a hardware unit that NVIDIA integrates into its GPU (graphics processing unit) to specifically process matrix calculations. TPU is a special hardware developed by Google that is also used to specifically process matrix calculations. In summary, in this embodiment, the tensor processing unit can be either an independent hardware (such as TPU) or a hardware unit located on a certain hardware (such as Tensor Core).

[0033] In the prior art, tensor processing units often process multiplication of sparse matrices and dense matrices, such as in graph neural networks. Taking graph neural networks as an example, in graph neural networks, the input of the graph neural network is a graph. The graph includes the graph structure and the properties of the graph nodes. The graph structure contains the connection relationship between the graph nodes. The properties of the graph nodes refer to the vectorized representation of the inherent properties of the graph nodes. The calculation of the graph neural network involves a large number of calculations between the properties of the graph nodes and the graph structure. Among them, the graph structure will be represented as a sparse matrix, such as Figure 2b is a graph structure, Figure 2a is the corresponding sparse matrix. The node properties of the graph are represented as a dense matrix. Figure 2b Taking the graph structure as an example, each node has a corresponding attribute value, and each node attribute value can be obtained as a 1*n vector after vectorization. Figure 2b There are 8 nodes in , then we can get a 8*n dense matrix in the end.

[0034] In the calculation of graph neural networks, the tensor processing unit is responsible for performing the above-mentioned sparse matrix and dense matrix operations. In the prior art, dense matrix operations (a conventional matrix calculation method, defined above) are generally used for sparse matrices and dense matrices. When performing dense matrix operations, the elements in the sparse matrix are multiplied in sequence with the elements in the corresponding rows or columns of the dense matrix. However, since there are many zero elements in the sparse matrix, no matter what the elements in the rows or columns of the corresponding dense matrix are, the final result is zero. Therefore, using the calculation method of dense matrix operations will bring unnecessary computing overhead in actual calculations, increasing storage costs and computing costs. Therefore, how to optimize the computing performance consumed by sparse matrix operations has become an important research direction in current deep learning and high-performance computing.

[0035] In order to optimize the operation of sparse matrices, the present application proposes a method for accelerating sparse matrix calculations on a tensor processing unit.

[0036] In the method for accelerating sparse matrix calculations on a tensor processing unit proposed in the present application, first, the sparse matrix to be calculated is reordered to increase the locality of the sparse matrix. Secondly, compression technology is used. By using five arrays, the original sparse matrix can be fully represented, thereby greatly reducing the corresponding memory usage. In addition, by optimizing the data loading method and allocating resources for thread block processing content in the tensor processing unit, the calculation of sparse matrices is accelerated. Preferably, compression technology can also be used. By using five arrays, the original sparse matrix can be fully represented, thereby greatly reducing the corresponding memory usage. For details, please refer to the attached Figure 1 .

[0037] Figure 1 FIG. 1 is a flow chart showing the main steps of a method for accelerating sparse matrix calculation on a tensor processing unit according to an embodiment of the present invention. Figure 1 As shown, the method for accelerating sparse matrix calculation on a tensor processing unit in an embodiment of the present invention mainly includes the following steps S10 to S40.

[0038] Step S10: Obtain a first matrix.

[0039] In this embodiment, the first matrix is ​​a sparse matrix. The matrix is ​​composed of a plurality of elements, wherein the elements are divided into zero elements and non-zero elements. The non-zero elements in the first matrix are referred to as non-zero elements.

[0040] In this embodiment, the first matrix is ​​obtained based on the input sequence. The input sequence is a kind of raw data. In some embodiments, taking the graph neural network as an example, the input sequence is the data input into the graph neural network, that is, the graph. The input sequence can be obtained through a data source or an input channel, and the data source can include but is not limited to data stored in the memory, an external database file, real-time sensor acquisition data, or a matrix data stream transmitted from the network.

[0041] After the first matrix is ​​obtained, the first matrix is ​​further processed through steps S20-S40.

[0042] Step S20: reorder the rows and columns of the first matrix to obtain a second matrix.

[0043] In this embodiment, reordering refers to rearranging the positions of the rows and columns in the first matrix according to certain rules. In the reordered matrix, the non-zero elements in the matrix are more concentrated, so that the locality of the first matrix is ​​better, and the cache hit rate is higher during operation.

[0044] In order to more easily understand the significance of the reordering of the present technology, the problems faced by sparse matrices in the prior art when performing calculations are explained here. The problem with sparse matrices during calculations is that they have poor locality and a low cache hit rate. Locality refers to the distribution characteristics of non-zero elements in rows and columns in a matrix. When locality is poor, the non-zero elements of the sparse matrix will be scattered throughout the matrix, resulting in frequent cross-row and cross-column data accesses during the calculation process. This reduces the memory access efficiency and the cache hit rate during the calculation process, resulting in low computing performance.

[0045] Cache Hit Rate refers to the success ratio of data being found directly from the cache when accessing the required data (a certain element or elements of the second matrix) in the cache system. In some embodiments, the cache is a shared memory. In matrix calculations, some commonly used elements that need to be used frequently are stored in the cache. If the required element is in the cache during the matrix access process, it will be determined whether the required element is in the cache. If the required element is in the cache, it is called a cache hit. If the required element is not in the cache, it is called a cache miss.

[0046] For example, if 80 out of 100 data requests hit the cache, the cache hit rate is 80%. However, the non-zero elements of sparse matrices are usually sparsely and irregularly distributed, resulting in discontinuity in data access. The design of the cache system relies on the principle of locality, which assumes that continuous data will be accessed frequently. But in sparse matrices, the distribution of non-zero elements is not concentrated and the locality is poor, so data access will jump between the memory and the cache. Therefore, the cache hit rate of sparse matrices is relatively low.

[0047] When a cache miss occurs during data access, data is then read from the memory to the cache for calculation. Reading data usually takes a lot of time (relative to the calculation process), so the low cache hit rate of sparse matrices in the prior art will extend the time spent on two matrix operations.

[0048] Therefore, in this step, by reordering the first matrix, the non-zero elements in the first matrix are concentrated as close to the diagonal as possible, thereby improving data locality. The optimized second matrix can make better use of the memory cache, improve the cache hit rate in subsequent operations on the second matrix, reduce the number of random accesses (reading data from global memory to shared memory), improve the efficiency of data reading and writing during the calculation process, and make subsequent matrix multiplication and other operations more efficient.

[0049] Furthermore, a row index value or a column index value of the first matrix is ​​obtained, and based on the row index value or the column index value, the rows and columns of the first matrix are reordered to obtain a second matrix.

[0050] The row index and column index are the values ​​used to locate a specific element in a matrix. They correspond to the rows and columns of the matrix, respectively.

[0051] The row index value refers to the number of the row in the matrix where an element is located. The row index usually starts at 0. Figure 2a As shown in the figure, the row index value corresponding to the first row of the matrix is ​​0, the row index value corresponding to the second row is 1, and so on. Similarly, the column index value refers to the number of the column where an element in the matrix is ​​located. By combining the row index value and the column index value, the element at a certain position in the matrix can be uniquely determined. Figure 2a For example, given [2,1], [2,1] represents the element at the 3rd row (row index is 2) and the 1st column (column index is 0). Figure 2a-2e and Figure 3 shown. Figure 2a-2e A schematic diagram of reordering a matrix in a specific scenario. Figure 3 2 is a schematic diagram of a reordering process. In one embodiment, through steps S201-S204, a second matrix with higher locality is obtained. Specifically, as follows:

[0052] Step S201: Perform node analysis on the first matrix to obtain the degree of the node and the connection relationship between the nodes.

[0053] In this embodiment, the node analysis refers to treating the row index value or column index value in the first matrix as a node in the graph, and constructing a graph structure representing the characteristics of the sparse matrix (first matrix) by analyzing the relationship between these nodes. The connection relationship between nodes refers to the connection between nodes, and the degree refers to the number of edges of the node after the node analysis.

[0054] In one implementation, the first matrix may be node-analyzed and a graph structure of the first matrix may be constructed in the following manner.

[0055] First, all non-zero elements of the first matrix are traversed, and the row and column index values ​​and numerical information of each non-zero element are recorded. Then, the row index value or column index value is used as a node to construct an initial node set.

[0056] In the process of node analysis, it is possible to determine whether there are non-zero elements in a row or column. When constructing the graph structure, the row index value or column index value corresponding to each row or column containing non-zero elements is used as a node in the graph. After that, the degree of each node is calculated. The specific operation is to count the connection relationship between the nodes to get the degree of the node.

[0057] Furthermore, the graph structure can be divided into two cases when it is represented: the first is a directed graph with in-degree and out-degree, which means that some nodes are connected to other nodes and can also receive connections from other nodes; the second case is an undirected graph without in-degree and out-degree, where the connection relationship between nodes is symmetrical, indicating that the association relationship between them is mutual, which is usually used to simplify the representation of matrix structure. In Figure 2, a symmetric matrix is ​​used, so the schematic diagram involving the graph structure is represented using an undirected graph.

[0058] Here we use an undirected graph with columns as nodes for illustration. Figure 2a and Figure 2b For example, Figure 2a is a first matrix in one embodiment, Figure 2b for Figure 2a The first matrix is ​​converted into a graph structure. As you can see, Figure 2a is an 8*8 matrix. In this 8*8 matrix, the row and column numbers are represented by digital symbols 0-7. Figure 2a In -e, the blank part represents zero elements and the solid part represents non-zero elements.

[0059] exist Figure 2aThere are three non-zero elements in the column with sequence number 1, which are P31, P41 and P61. The sequence numbers of the rows corresponding to these three non-zero elements are 3, 4 and 6 respectively. Therefore, when forming the graph structure, the node with sequence number 1 (hereinafter referred to as node 1) will be connected to the three nodes with sequence numbers 3, 4 and 6. Observe Figure 2b It can be seen that node 1 is indeed connected to the other three nodes.

[0060] Similarly, let's take the example of column 4. There are five non-zero elements in column 4. These five non-zero elements are P04, P14, P24, P34, and P74. The numbers of the rows corresponding to these five non-zero elements are 0, 1, 2, 3, and 7. Therefore, when forming the graph structure, node 4 will be connected to nodes with numbers 0, 1, 2, 3, and 7. Observation Figure 2b It can be seen that node 4 is indeed connected to the other five nodes.

[0061] The graph structure in the above is an undirected graph. If the graph structure is a directed graph, a directed graph can be obtained by using only columns as nodes and then rows as nodes. It can be seen that directed graphs are also applicable to the above technical solution. Therefore, the technical solution does not limit the graph structure to a directed graph or an undirected graph.

[0062] By repeating the above method, the connection relationship between nodes can be determined, and then the degree of the node can be determined. Then, based on the connection relationship between the nodes and the degree of the node, the rows and columns of the first matrix are reordered to obtain the second matrix. Specifically, steps S202-S204 are as follows:

[0063] Step S202: merging the nodes based on the sizes of the degrees, and expanding the merged nodes to obtain a first array.

[0064] In this embodiment, through this merging and expanding method, nodes with a closely connected relationship can be effectively organized together, thereby optimizing the node arrangement structure of the entire sparse matrix.

[0065] In one embodiment, the merging and expanding methods are different. Specifically, in the merging process, the connectivity increment is used as a condition for determining whether the merging is possible. The connectivity increment represents the priority of the node merging order. In the expanding process, the merged nodes are expanded by depth-first traversal.

[0066] exist Figure 2cIn the example, the merged nodes are expanded in a tree diagram, and the nodes at the bottom of the tree diagram after expansion are called leaf nodes. A series of arrays consisting of nodes are obtained along the direction of the leaf nodes, namely the first array. The first array can be converted into a matrix after processing, and the locality of the matrix converted from the first array is better than the locality of the first matrix. In this way, the original scattered data can be re-sorted according to the connection increment and the expansion rules, so that the nodes with closer connections are closer, and finally the locality is increased.

[0067] refer to Figure 2c , Figure 2c The expanded diagram of the merged nodes is shown in Figure 2. Figure 2c In this embodiment, the first array is obtained through three sub-steps, as follows:

[0068] (1) sorting the nodes according to the sizes of the degrees;

[0069] by Figure 2b Take for example, where the degrees of each node are shown in Table 1:

[0070]

[0071]

[0072] Table 1

[0073] The nodes are sorted according to a preset size arrangement rule, thereby obtaining an array, which is called the third array. The preset size arrangement rule here is to arrange the corresponding degrees from small to large.

[0074] In this implementation, one of the arrangements is selected, and the resulting third array is as follows: 5, 6, 7, 1, 2, 3, 0, 4.

[0075] (2) merging the nodes along the order of the third array until a preset merging condition is met;

[0076] In this embodiment, two nodes are merged to form a new node. The purpose of merging is to effectively group nodes with close connections and lay the foundation for subsequent reordering. Specifically, a method for merging nodes is provided, that is, merging by increasing the degree of connectivity, as follows:

[0077] First, along the third array, the connection degree increments between adjacent nodes are calculated and obtained.

[0078] The calculation formula of the connectivity increment is as follows:

[0079]

[0080] Where m is the total number of edges in the graph; i and j are nodes; A ij : represents the weight between node i and node j (if there is an edge between i and j, then A ij is 1, otherwise 0); k i : represents the degree of node i; Indicator function, when node i and node j are in the same community, the function value is 1; otherwise it is 0.

[0081] In this embodiment, the connectivity increment ΔQ measures the change in connectivity after merging two nodes into the same community. If ΔQ>0, it means that merging the two nodes together can enhance the community structure. On the contrary, if ΔQ≤0, it is unreasonable to merge the two nodes and they cannot be merged.

[0082] A community is a subset of nodes in a graph structure. The connections between these nodes in the same community are denser than the connections with other nodes in the graph. In other words, a community is a group of nodes in a graph structure that are closely connected internally and sparsely connected externally.

[0083] In addition, as to whether two nodes are in the same community, algorithms can be used to detect whether they are in the same community. For example, the Louvain algorithm, the Girvan-Newman algorithm, the Kronecker Delta function, etc. can all be used to determine whether two nodes are in the same community.

[0084] refer to Figure 2b , take nodes 5 and 6 as an example. Since nodes 5 and 6 are not directly connected, their A 56 is 0, and we can directly deduce that the connection increment ΔQ ≤ 0 between the two, and node 5 and node 6 cannot be directly merged.

[0085] However, for example, whether nodes 3 and 4 should be merged, whether the new nodes after the merger can be merged with the new nodes, and whether the new nodes can be merged with the nodes. At this time, it is necessary to determine whether the two nodes to be merged are in the same community and whether the connection increment is greater than 0. It can be seen that the prerequisite for the connection increment to be greater than 0 is that the two nodes must be in the same community and whether there is a direct connection between the two nodes. When one of these two conditions is not met, the merger cannot be performed.

[0086] The above requirements also apply to the further merging of the merged new nodes. Whether the merged new nodes can be merged with new nodes or with nodes is also determined by the connection increment. At this time, the merged new nodes are treated as one node for processing.

[0087] Furthermore, if the connection increment between a node and multiple nodes is greater than 0 at the same time, it means that the node can be merged with multiple nodes. At this time, the connection increment is used to determine which two nodes are to be merged. The connection increment is not only a basic condition for determining whether two nodes can be merged, but also represents the priority of the node merging order. In this embodiment, when the connection increment between a node and multiple nodes is greater than 0, the node with the highest connection increment is selected for merging.

[0088] Then, the nodes are merged layer by layer according to the connectivity increment until the number of merged nodes is the same as the number of communities in the first matrix. According to the connectivity increment formula given above, nodes that are not in the same community cannot be merged. The layer-by-layer merging here refers to the merging between new nodes and new nodes and the merging between new nodes and nodes. The final node may contain multiple nodes, that is, it has undergone multiple mergers.

[0089] Reference for the effect after merging Figure 2c As shown, Figure 2c This is a schematic diagram of the nodes after merging, where each layer of the tree diagram records the merging of nodes. Figure 2c It can be seen that nodes 5, 0, 7, 2, and 4 belong to the same community, and these five nodes are finally merged into one root node. Nodes 6, 1, and 3 belong to another community, and these three nodes are finally merged into another root node. Specifically, during the node merging process, node 5 and node 0 were merged to obtain a new node 0'. Node 7 and node 2 were merged to obtain a new node 2'. Node 2' and node 4 were merged to obtain a new node 2". Finally, node 2" and node 0' were merged to obtain the root node 2"'. Similarly, another community was merged to obtain the root node 1"'. These two root nodes belong to different communities and can no longer be merged.

[0090] Finally, the merged nodes are expanded in a tree diagram to obtain a first array, wherein the first array refers to a node set formed along the direction of leaf nodes of the tree diagram.

[0091] like Figure 2c As shown, the node set (first array) formed by the directions of the leaf nodes of the tree diagram obtained in one expansion mode is 5, 0, 7, 2, 4, 6, 1, 3. In other words, the arrangement of the leaf nodes of the tree diagram is related to the expansion mode.

[0092] In one embodiment, the merged nodes are expanded using a depth-first traversal (DFS) method. In this embodiment, the depth-first traversal can deeply traverse closely connected nodes to maintain the relevance between nodes. Through this expansion method, the reordered non-zero elements can be more concentrated, improving the locality of the corresponding first array matrix.

[0093] Step S203: modifying the first array based on the connection relationship between the nodes to obtain a second array.

[0094] In this embodiment, the first array is modified to obtain the second array. The purpose of the modification is to further improve the locality of the matrix after reordering. The nodes in the first array are arranged in a first order, and the nodes in the second array are arranged in a second order. The first order and the second order are orders obtained according to different rules.

[0095] In one implementation, the second array is obtained through steps S203-1 to S203-3, as follows:

[0096] Step S203-1: Determine the node with the first priority.

[0097] In this embodiment, the sequence is the basis for subsequent correction, and the second array corrected later is obtained based on the sequence. At the same time, the sequence of the nodes is related to the position order in the first array, and the specific analysis will be detailed later.

[0098] In one embodiment, the first node obtained by using the depth-first traversal algorithm is used as the first-order node. In another embodiment, the node with the smallest degree is used as the first-order node. The first-order nodes obtained by these two embodiments are the same.

[0099] It should be noted that in the above description, the first-order node is determined in order to find the starting point of the connection relationship, so the node with the smallest degree is selected as the first-order node. It can be seen that, on the contrary, the end point of the connection relationship is selected in the same way, that is, the node with the largest degree can be found as the first-order node. This will only affect the order of the size arrangement of the sequence, and will not affect the final locality.

[0100] Step S203-2: Based on the connection relationship between the nodes, determine another node that is closest to the node of the first priority in the second-order relationship, and obtain a node with a second priority.

[0101] In this embodiment, the second-order relationship refers to a node that has a common neighbor node with the first-order node.

[0102] by Figure 2bFor example, assume that the first-order node is node 5. Then the neighbor node of the first-order node is node 0. The common neighbor nodes of the first-order node are node 2, node 4, and node 7. These three nodes all have a common neighbor node (node ​​0) with node 5, so these three nodes form the second-order relationship of the first-order node.

[0103] The same method is used to find the second-order relationship of node 0. The neighbor nodes of node 0 include node 2, node 4, node 5, and node 7. At this time, there are no other nodes that have exactly the same neighbor nodes as node 0, but there are nodes that have some of the same neighbor nodes as node 0, such as node 2, node 7, node 4, node 3, and node 1. These nodes all have at least one node in common with node 0, so these five nodes constitute the second-order relationship of node 0.

[0104] In this embodiment, another node closest to the first-ranked node in the second-order relationship is found. In this embodiment, the closest node in the second-order relationship refers to the node with the most neighbor nodes in the second-order relationship with the first-ranked node.

[0105] Still taking node 5 as the first-ranked node as an example, Figure 2b The connection relationship between the nodes can be obtained, and then it can be found that node 7, node 4 and node 2 all have the same number of neighbor nodes as node 5.

[0106] At this time, if the nearest node in the second-order relationship is not unique, the node with the highest ranking in the first array is used as another nearest node in the second-order relationship. In the first array, node 7 is ranked the highest, so the second-ranked node obtained at this time is node 7.

[0107] If there is only one node in the second-order relationship that has the most neighbor nodes with the first-order node, then this node naturally becomes the second-order node.

[0108] Step S203-3: Determine another node closest to the second-order relationship of the 1+Nth-ranked node, obtain the 2+Nth-ranked node, and obtain the sequence of all nodes in the first array.

[0109] In this embodiment, N is a positive integer.

[0110] In one embodiment, when searching for the nearest node in the second-order relationship, the second-order relationship does not include nodes that have already been ranked. That is, when searching for the third-ranked node, the second-order relationship no longer includes the second-ranked or first-ranked nodes.

[0111] In this embodiment, after the node with the second priority is determined, the nodes with the third priority, the fourth priority, and so on are determined in the same way until the priority order of all nodes in the first array is determined.

[0112] Step S203 - 4 : Arrange the nodes based on the sequence to obtain a second array.

[0113] refer to Figure 2c , where the numbers below the first array are the corrected second array.

[0114] by Figure 2c For example, the original node order of the first array is 5, 0, 7, 2, 4, 6, 1, 3. After correction, the node order of the second array is 5, 7, 0, 2, 4, 6, 1, 3. It can be seen that after correction, node 7 is located before node 0. However, the method for generating the first array uses the node traversal method. After searching for node 5, it will return to node 0' and then search for node 0. Or after searching for node 7, it will return to node 2' and then search for node 2. Therefore, the method for generating the first array cannot fully consider the connection relationship between nodes like the second array.

[0115] Because the connection relationship between nodes is more fully considered in the process of modifying the first array, the matrix formed by the second array will have better locality and a higher cache hit rate during calculation.

[0116] Step S204: performing inverse node conversion based on the connection relationship between the second array and the nodes to obtain a second matrix.

[0117] In this embodiment, the rearranged second array is converted into a matrix again to obtain a second matrix.

[0118] In one implementation, the rearranged second array is reconstructed in the form of a matrix to obtain a second matrix. Specifically, the nodes in the second array can be mapped back to the matrix structure according to the arrangement order of the nodes, so that the connection relationship between the nodes is presented in a more optimized manner in the new matrix, ensuring that the non-zero elements are as concentrated as possible in the matrix, thereby improving the efficiency of matrix operations.

[0119] First, according to the preset mapping rules, the nodes in the second array are mapped one by one to form a fourth array. In this mapping process, the position of each node will be re-mapped to the fourth array according to its arrangement order in the second array, ensuring that the association relationship of the nodes is completely preserved and mapped.

[0120] Subsequently, the fourth array is constructed into a third matrix in a matrixed manner, maintaining the connection relationship and arrangement order between nodes. The resulting third matrix will present the structure of the original sparse matrix in a more compact and optimized manner, ensuring that non-zero elements are concentratedly distributed.

[0121] In one example, refer to Figure 2c , Figure 2d and Figure 2e .in Figure 2c The bottom-level data is the fourth array, which can be seen to be obtained by mapping the second array in sequence. Figure 2d is the graph structure obtained through the third array. By processing this graph structure, we can get Figure 2e The matrix in .

[0122] observe Figure 2e The matrix in Figure 2a The first matrix in Figure 2e The locality of the second matrix is ​​greatly improved, so that the memory cache can be better utilized, the cache hit rate in subsequent operations is improved, the number of random accesses is reduced, the efficiency of data reading and writing in the calculation process is improved, and subsequent matrix multiplication and other operations are more efficient.

[0123] Step S30: compressing the second matrix in a preset format, and storing the compressed second matrix in a preset storage space for compression.

[0124] In this embodiment, the compressed second matrix is ​​used to save memory usage and improve the efficiency of memory access. The preset storage space is a pre-set storage space, and the preset storage space is used to store the compressed second matrix for easy access during calculation.

[0125] In some embodiments, the preset storage space may be a memory, a hard disk, a disk, a cache, or a variety of similar storage media, or a partition of the above storage media. In this embodiment, the compressed second matrix is ​​stored in the global memory.

[0126] The second matrix is ​​compressed in a preset format, and the preset format is the format used for compression. In the present technology, five arrays are used to represent the second matrix. The arrays here are different from the first array or the second array mentioned above. The array obtained after compression here is a combination of numbers, which represent the specific content of the second matrix, while the array in the above is a collection of nodes.

[0127] The specific compression method is performed through steps S301-S303, which are as follows:

[0128] Step S301: dividing the second matrix into M windows according to a preset first division rule.

[0129] In this embodiment, the M windows are connected in sequence.

[0130] In one implementation, the second matrix is ​​evenly divided into M windows. Different windows have the same number of rows and columns. The first segmentation rule may be to divide the number of rows of the second matrix equally, thereby dividing a number of regions with the same number of rows, each region being called a window, wherein the number of columns of the window is the number of columns of the second matrix. The M windows are sequentially connected to form the entire second matrix.

[0131] Furthermore, in one implementation, the second matrix may be preprocessed before being segmented, such as zero padding. For example, when the second matrix is ​​a 15*15 matrix, the second matrix may be padded with columns, and the padded elements are 0, so that the second matrix becomes a 16*16 matrix, which is easier to be segmented.

[0132] refer to Figure 4 , Figure 4 On the left is the second matrix. Figure 4 The second matrix shown in FIG is a 16*16 matrix. Figure 4 In the example, the second matrix is ​​divided into 4 windows ( Figure 4 in the Row Window).

[0133] Step S302: performing block segmentation on the window based on non-zero elements to obtain a segmentation result consisting of matrix blocks.

[0134] In this embodiment, the matrix block is not only used as a segmentation result, but also serves as a basic unit for subsequent tensor processing units to process.

[0135] In one embodiment, the window is segmented according to the non-zero elements in the window, and only the columns where the non-zero elements in the window are located are taken as the columns of the matrix block. In this embodiment, the number of rows and columns of the matrix block is equal. Among them, the number of rows of the matrix block is related to the number of rows of the window in which it is located. In some embodiments, the number of rows of the matrix block is the same as the number of rows of the window in which it is located. For example, a window is a 4*16 matrix (4 rows and 16 columns), and when block segmentation is performed, the rows of the matrix block are also 4 rows. If the number of rows and columns of the matrix block is the same, the matrix block is a 4*4 matrix.

[0136] When the window is segmented into blocks, if the number of columns where non-zero elements are located in a window is less than the number of rows, the columns of the matrix block are padded with zeros.

[0137] Still Figure 4For example, you can see that Figure 4 Although the first window has 16 columns, only 12 columns have non-zero elements, and the number of rows in the window is 4, so the 12 columns where the non-zero elements are located in the window are divided into 3 matrix blocks (TCblock). The dark columns in the figure are the columns of the matrix blocks. Each matrix block has only 4 columns, the same as the number of rows. It can be seen that when the window is divided into matrix blocks, the columns of the matrix blocks can be jumped and are not required to be continuous. Figure 4 In the second window, only 3 columns have non-zero elements. However, when the second matrix is ​​segmented, all matrix blocks are required to have the same rows and columns. Therefore, the matrix blocks in the second window need at least 4 columns. Therefore, a column is supplemented with zero elements so that the matrix blocks in the second window still have the same rows and columns as the matrix blocks in other windows, which is convenient for subsequent processing.

[0138] Depend on Figure 4 It can be seen that the number of matrix blocks in each window is different. Secondly, the column indexes of the second matrix corresponding to the matrix blocks are not necessarily continuous. Figure 4 Take the matrix block of the second window in as an example. The second window has only one matrix block, and its column indexes are 0, 7, 9, and 10. Therefore, the column indexes of the matrix blocks are different, and the column index can help the subsequent tensor processing unit to quickly locate which column in the second matrix has non-zero elements.

[0139] After the matrix block division is completed, step S303 is executed.

[0140] Step S303: Based on the segmentation result, obtain the window matrix block offset, the previous non-zero element offset of the matrix block, the matrix block column index, the non-zero element position set of the matrix block and the non-zero element numerical sequence, wherein the non-zero element numerical sequence is obtained based on the non-zero elements in the matrix block.

[0141] As described above, five arrays are used to represent the second matrix. The five arrays are window matrix block bias, previous non-zero element bias of the matrix block, matrix block column index, non-zero element position set of the matrix block, and non-zero element numerical sequence. The five arrays here are digital combinations obtained after the second matrix is ​​compressed, and these digital combinations represent the specific content of the second matrix, and the first array, the second array, or the third array in the above represents a set of nodes.

[0142] These five arrays are described in detail as follows:

[0143] (1) Window matrix block bias:

[0144] The window matrix block offset consists of two parts, namely the previous matrix block count of the window and the total number of matrix blocks in the second matrix.

[0145] The previous matrix block count of a window refers to the number of matrix blocks in the window before the window. Figure 4 For example, the first window has no previous window, so there is no previous matrix block. Therefore, the previous matrix block count (RowWindowOffset) of the first window is 0. The second window has three matrix blocks first, so the previous matrix block count of the second window is 3. The total number of matrix blocks in the second matrix is ​​7, so the final number is 7.

[0146] The reason for designing the window matrix block bias in this way is that the thread block processing in the tensor processing unit processes the matrix according to the window. In one embodiment, each thread block can process one window, and multiple thread blocks can process multiple windows, thereby realizing parallel calculation of the second matrix. The minimum unit of the thread block when processing the data in the window is the matrix block. Therefore, the window matrix block bias can clearly indicate that the second matrix is ​​divided into several windows, and there are several matrix blocks in each window, which is convenient for subsequent processing.

[0147] (2) The previous non-zero element bias and non-zero element numerical sequence of the matrix block:

[0148] The previous non-zero element bias of a matrix block includes two parts, the previous non-zero element count of the matrix block and the total number of non-zero elements in the second matrix.

[0149] The counting rule of the previous non-zero elements of a matrix block is similar to the counting rule of the previous matrix block of a window. Figure 4 As shown, the first matrix block is the first matrix block of the first window, and it has no non-zero elements before, so it is recorded as 0. Next, take the fourth matrix block as an example, the fourth matrix block is located in the second window. It has 14 non-zero elements before, so it is recorded as 14. And so on, and finally add the total number of non-zero elements in the second matrix to form the previous non-zero element bias of the matrix block. It can be seen that the previous non-zero element bias of the matrix block can find out how many non-zero elements a matrix block has before, and can find out how many non-zero elements the matrix block has (just subtract the previous non-zero element bias of the next matrix block).

[0150] In this embodiment, the previous non-zero element bias of the matrix block works together with the non-zero element numerical sequence. The non-zero element numerical sequence is obtained based on the non-zero elements in the matrix block. In other words, the non-zero element numerical sequence is the result of recording the non-zero elements in the matrix block one by one in order, taking the matrix block as a unit.

[0151] The previous non-zero element bias of the matrix block can quickly lock which non-zero elements in the non-zero element numerical sequence are non-zero elements in the matrix block. And quickly lock the position in the non-zero element numerical sequence, so as to directly extract these non-zero elements. Because when extracting non-zero elements, first extract in units of matrix blocks (corresponding to the window matrix block bias), and then extract in units of columns (corresponding to the previous non-zero element bias and non-zero element numerical sequence of the matrix block), so that in the extraction process, the zero elements in the extracted area are reduced layer by layer. The reduction of zero elements, in turn, reduces the interference caused by zero elements in the extraction process, and ultimately improves the cache hit rate.

[0152] (3) Matrix block column index:

[0153] The matrix block column index refers to the index of the column of the second matrix where the non-zero elements in the matrix block are located. Take the first matrix block in the first window as an example, where the column index of the matrix block is 0, 1, 2, 3. Through the column index, the matrix block can be quickly located on the column of the window where it is located.

[0154] (4) The set of non-zero element positions of the matrix block:

[0155] The non-zero element position set of a matrix block refers to the position arrangement of the non-zero elements of the matrix block itself.

[0156] In one implementation, a one-dimensional array is used to represent the positions of non-zero elements in a matrix block. Figure 4 For example, there are 5 non-zero elements and 11 zero elements in the first matrix block in the first window. Then the non-zero element position set of the matrix block can be represented by "0011010001001000", where 1 represents a non-zero element and 0 represents a zero element.

[0157] The non-zero positions of the matrix block can be stored through the non-zero element position set of the matrix block, saving matrix space. Furthermore, by compressing the above one-dimensional array, a shorter mathematical notation can be obtained, thereby making the non-zero element position set of the matrix block more concise.

[0158] If we use uint64 integer to change "0011010001001000", we get 4632 ( Figure 4 The length of the data is compressed.

[0159] In this embodiment, the second matrix is ​​compressed by using five arrays, namely, window matrix block bias, matrix block prior non-zero element bias, matrix block column index, matrix block non-zero element position set, and non-zero element numerical sequence, which greatly reduces the number of characters and improves the compression rate. This reduces the memory usage and improves the memory access efficiency. After testing, the compression method using step S30 improves the compression efficiency by about 40% compared to the traditional SOTA.

[0160] Step S40: read the third matrix from the memory of the GPU, and perform operations on the second matrix and the third matrix.

[0161] In this embodiment, the third matrix refers to a matrix that operates with the second matrix, and the third matrix is ​​a dense matrix. The third matrix is ​​obtained based on the input sequence. In the previous description, the graph neural network has been used as an example, and will not be repeated here.

[0162] The GPU memory here refers to the global memory.

[0163] Afterwards, step S40 will be divided into two parts for detailed description. One is the process of reading and operating the third matrix, and the other is how to schedule and allocate on the tensor processing unit to realize the process of operating the second matrix.

[0164] Among them, the process of "reading the third matrix and performing operations" is a periodic repetitive process. In the training of a neural network, a large number of sparse matrices (second matrices) and dense matrices (third matrices) are involved. However, in the specific processing process, it is generally processed pair by pair, that is, one second matrix and the corresponding third matrix are processed each time. In the process of processing a large number of matrices in pairs, a periodic process will be formed. Therefore, the process of reading the third matrix and performing operations is a periodic and continuous repetitive process until all the matrices to be processed are processed.

[0165] refer to Figure 5 , Figure 5 This is a schematic diagram of reading and operating on a matrix. Figure 5 In this context, loading refers to completing a loading action, such as loading from global memory to shared memory or register memory.

[0166] Reading and operating the matrix can be divided into a preprocessing process and a subsequent periodic repetition process. Figure 5The preprocessing part and the three cycles after the preprocessing are included. The preprocessing is used to provide calculation data for the subsequent synchronization cycle. The repetitive process is composed of the synchronization cycle as the smallest unit. The synchronization cycle includes multiple operations, and a synchronization cycle starts with a synchronization operation and ends with a synchronization operation. The most intuitive difference between the synchronization cycle and the preprocessing is that the synchronization cycle contains the operation of the second matrix and the third matrix. So, Figure 5 As shown, the first synchronization cycle includes the first operation of the second matrix and the third matrix (the first matrix operation). Therefore, the preprocessing should at least include: reading the second matrix and the third matrix required in the first synchronization cycle operation (loading the first third matrix and loading the first second matrix).

[0167] Furthermore, the reading of the second third matrix depends on the corresponding matrix block column index, where the matrix block column index is one of the five arrays after the second matrix is ​​compressed. Therefore, the preprocessing also includes: reading the matrix block column index corresponding to the third matrix required in the first synchronization cycle operation (loading the first matrix block column index).

[0168] Furthermore, since each synchronization cycle involves a calculation operation, the second synchronization cycle will also involve a calculation operation, so the second matrix and the third matrix required in the calculation operation of the second synchronization cycle will be read in the first synchronization cycle. As can be seen from the foregoing content, the reading of the third matrix is ​​based on the matrix block column index. Therefore, the preprocessing before the first synchronization cycle also includes: loading the matrix block column index corresponding to the third matrix required in the calculation operation of the second synchronization cycle (loading the second matrix block column index).

[0169] In summary, the preprocessing includes: loading the first matrix block column index, the second matrix block column index, loading the first third matrix, and loading the first second matrix.

[0170] Summarizing the above process, we get the following:

[0171] Step S40 includes: performing preprocessing, and after the preprocessing, performing repeated operations with the synchronization cycle as the minimum unit, until the synchronization cycle satisfies: in the x-th synchronization cycle, loading the n-1th third matrix into the register memory; loading the n-th matrix block column index into the shared memory; loading the n-1th second matrix into the shared memory, and calculating the n-2th matrix operation, wherein the n-2th matrix operation is obtained based on the n-2th second matrix and the n-2th third matrix, wherein loading the n-1th third matrix is ​​based on loading the n-1th matrix column index in the x-2th synchronization cycle, and x≥3.

[0172] Likewise Figure 5As shown, each synchronization cycle depends on the matrix block column index loaded two synchronization cycles before, and also depends on the second matrix and the third matrix required for the current matrix operation loaded in the previous synchronization cycle.

[0173] like Figure 6 As shown, Figure 6 It is a timing diagram for reading and calculating a matrix. The dotted line part is the synchronization operation (sync). In one implementation, taking the tensor processing unit as Tensor Core as an example, the Tensor Core requires the following caches in the GPU during the calculation process, namely L2 cache, L1 cache, shared memory and register memory. The reading speed of each level gradually increases. At the same time, unlike the gradual increase in the reading speed, the storage space itself becomes smaller step by step.

[0174] like Figure 6 As shown in the figure, GToSHM, which stands for Global to Shared Memory, is from global memory to shared memory. GToReg, which stands for Global to Register, is from global memory to register memory. TCMMA represents the calculation operation of the matrix.

[0175] In matrix calculations, the matrix block column index is used to find the corresponding row to be calculated in the third matrix. Since the calculation is performed in units of matrix blocks, it involves repeated calls to the column data (matrix block column index) of the original matrix where the matrix block is located. Therefore, in order to reduce the number of calls, the matrix block column index is loaded from the global memory to the shared memory. During calculation, the Tensor Core needs to take the third matrix to the register memory according to the matrix block column index, so as to quickly and accurately extract the data at the corresponding position of the third matrix from the register memory during the operation. In summary, the prerequisite for obtaining the third matrix is ​​to obtain the corresponding matrix block column index.

[0176] When the matrix operation (TCMMA) has not yet started, the previous data is preprocessed and no loop processing has been formed. Figure 6As shown, when the matrix operation has not yet started, the 0th matrix block column index (GToSHMAToB[0]) and the 0th second matrix (GToSHM sparseA[0]) in the input data do not need to follow a strict order, that is, there is no strict order between the operations in the preprocessing. However, when the matrix operation starts, while calculating in a synchronization cycle (TCMMA0), it is necessary to load the matrix block column index (GToSHMAToB[2]) to the shared memory at the same time, and read the third matrix (GToReg dense B[1]) at the same time. The third matrix (GToReg dense B[1]) read in this synchronization cycle is read according to the corresponding matrix block column index loaded into the shared memory in the previous synchronization cycle (GToSHM AToB[1]).

[0177] It can be seen that through this technology, when performing a calculation (TCMMA0), the second matrix (GToSHM sparseA[1]) and the third matrix (GToReg dense B[1]) used for the next calculation can be loaded, and at the same time, the corresponding matrix block column index can be recorded in preparation for the next calculation.

[0178] By using the above method, parallel operations are performed within one synchronization cycle. The matrix loading required for the next matrix operation can be completed while the matrix operation is being performed. This reduces the waiting time between two matrix operations and speeds up the calculation of sparse matrices.

[0179] In addition, such a design allows the second matrix and the third matrix loaded in the current synchronization cycle to be calculated in the next synchronization cycle, thereby realizing fast data entry and exit and reducing the occupancy of shared memory and register memory.

[0180] Furthermore, the tensor processing unit also includes a thread block, wherein the thread block is used to perform matrix calculations. In the prior art, each thread block is only responsible for the calculation of all matrix blocks in a window. Therefore, the matrix block is the smallest unit in thread block processing. As can be seen from the above content, the number of matrix blocks in a window can be more or less, which is extremely unbalanced, so the number of thread blocks processed by each matrix block is also more or less, which is very unbalanced.

[0181] Therefore, in one embodiment, the tensor processing unit operates on the compressed matrix and the third matrix, including: based on the data retrieval time, calculation time and write-back time, the original tasks are reallocated so that the time for the thread blocks to finally process data is as equal as possible.

[0182] The fetch time refers to the time to load the matrix (compressed matrix or third matrix) from the global memory or shared memory to the thread block, the compute time refers to the time the thread block actually executes and computes a single matrix operation, and the write-back time refers to the time to write the calculation result back to the global memory. Generally speaking, for a task, the compute time is the shortest, the fetch time is the second, and the write-back time is the longest.

[0183] This step determines the workload distribution of each thread block in the original task by comprehensively analyzing and counting these three times. The original task is the task in each thread block before load balancing is performed. The original task is divided into three operations, namely, data acquisition operation, calculation operation and write-back operation. However, from another perspective, the time consumed by the original task is directly proportional to the matrix block to be processed. One matrix block to be processed will bring one data acquisition operation and one calculation operation.

[0184] In the process of reallocating tasks, we first calculate the data acquisition time, calculation time, and write-back time of the corresponding task for each thread block, and sum them up to get the total processing time used by the thread block to process the original task. Then, based on the total processing time of the thread block, we redivide the tasks, and allocate the tasks with larger workloads to other thread blocks with smaller total processing time (lighter load), to ensure that the total processing time of each thread block is as balanced as possible when performing calculations. This adjustment method can make full use of the computing power of each thread block and avoid the overall efficiency reduction caused by too many tasks in some thread blocks.

[0185] It should be noted that when the thread block processes the original task, only one write-back is used, that is, the total processing time for processing the original task only accounts for one write-back time. However, when the thread block processes other tasks, the number of write-backs may increase. When the thread block processes a task, the number of write-backs is the same as the number of windows corresponding to the processed matrix block.

[0186] In other words, if the matrix block processed by a thread block comes from only one window (the case of the original task), only one write-back operation is required, and the total processing time also has only one write-back time. However, if the matrix block processed by a thread block comes from two windows, the thread block will perform two write-back operations in the overall task, and there will be two write-back times in the total processing time. The number of fetching time and computing time is the same throughout the entire process of a thread block processing. The number of fetching operations and computing operations will be the same as the number of matrix blocks processed.

[0187] So the total processing time (T) of a thread block is calculated as:

[0188] T=tcm*(lot+cot)+rwm*wt

[0189] Among them, tcm is the number of matrix blocks processed, lot is the single data acquisition time, cot is the single calculation time, rwm is the number of windows corresponding to the processed matrix block, and wt is the single write back time.

[0190] Then, the total processing time of each thread block in the original task is calculated according to the above calculation method, and then the total processing time is analyzed. The single read time, single calculation time and single write back time are all preset.

[0191] In one implementation, if the total processing time difference between the thread blocks is relatively large, load balancing can be performed. After load balancing is performed, the total processing time of each thread block is recalculated. At this time, due to load balancing, the total processing time difference between the thread blocks will be reduced and will be less than the longest total processing time before load balancing is performed. In other words, the distribution of the total processing time after redistribution is more consistent with the total processing time of the original task.

[0192] Specifically, in this embodiment, the reallocated tasks are obtained by a mathematical model. This model has two objectives, one is a primary objective and the other is a secondary objective. The primary objective is to minimize the difference in processing time of all thread blocks so that the total processing time distribution is more balanced. The secondary objective is to ensure that the maximum value of the total processing time after reallocation is less than the maximum value of the total processing time in the original task. The primary objective is achieved through an objective function, and the secondary objective is achieved through constraints or penalty items.

[0193] After that, the constraints are specified and the process is optimized, and finally the optimal new task arrangement of the thread block is obtained. Then, new tasks are arranged for each thread block according to the new task arrangement.

[0194] refer to Figure 7 ,in Figure 7 Part (a) of Figure 1 is a schematic diagram before load balancing, where the left figure shows the number of matrix blocks processed by different thread blocks, and TB0-4 represents five thread blocks. In the left figure, the matrix blocks processed by each thread block are from the same window. In other words, Figure 7 Part (a) in the middle shows the state of processing the original task. In contrast, the right figure of part (a) shows the time taken by each thread block to process the task on the left.

[0195] Figure 7Part (b) shows the schematic diagram after load balancing. The dark-colored blocks in the left figure indicate that the matrix blocks processed by the current thread block come from different windows. Take TB0 and TB1 as examples. The blocks in TB0 are all light-colored, indicating that the matrix blocks processed in TB0 are all from one window. One of the two matrix blocks of TB1 is light-colored and the other is dark-colored (pure black), indicating that the two matrix blocks of TB1 come from two different windows. It should be noted that there is no relationship between the light-colored blocks in TB0 and the light-colored blocks in TB1, and they can be from the same window or from different windows.

[0196] It can be seen that the total processing time of a thread block consists of three parts, which are consistent with the above formula and will not be repeated here. After load balancing, the total processing time of each thread block is very uniform, and the time for processing data is as equal as possible, so that the load between thread blocks is more balanced and the waiting phenomenon is reduced.

[0197] By estimating the time it takes a thread block to complete a single operation, we get a way to calculate the total processing time of the thread block, and based on that we design the objective function, and finally get a better matrix block task allocation solution. This effectively redistributes the tasks of the thread block, achieves load balancing, and ensures that the data fetching, computing, and writing back operations can work more closely together. This step maximizes the use of computing resources by adjusting the task allocation of the thread block, reduces the waiting time between thread blocks, and significantly improves the overall performance and efficiency of the tensor processing unit when performing sparse matrix calculations.

[0198] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0199] It is understood by those skilled in the art that all or part of the processes in the method for implementing any of the above embodiments of the present invention can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of each of the above method embodiments when executed by the processor. Among them, the computer program includes a computer program, and the computer program can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0200] Furthermore, the present invention also provides a control device. In an embodiment of a control device according to the present invention, the control device includes a processor and a storage device, the storage device can be configured to store a program for executing a method for accelerating sparse matrix calculations on a tensor processing unit of the above method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to a program for executing a method for accelerating sparse matrix calculations on a tensor processing unit of the above method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The control device can be a control device device formed by various electronic devices.

[0201] Furthermore, the present invention also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present invention, the computer-readable storage medium can be configured to store a program for executing a method for accelerating sparse matrix calculations on a tensor processing unit of the above-mentioned method embodiment, and the program can be loaded and run by a processor to implement the method for accelerating sparse matrix calculations on the above-mentioned tensor processing unit. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-temporary computer-readable storage medium.

[0202] Further, it should be understood that since the setting of each module is only for illustrating the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the software in the processor, a part of the hardware, or a part of the combination of software and hardware. Therefore, the number of each module in the figure is only schematic.

[0203] Those skilled in the art will appreciate that the modules in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principle of the present invention, and therefore, the technical solutions after splitting or merging will fall within the protection scope of the present invention.

[0204] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for accelerating sparse matrix calculations on a tensor processing unit, characterized in that: include: Obtain a first matrix, wherein the first matrix includes row index values ​​or column index values, and the first matrix is ​​a sparse matrix obtained based on an input sequence; Based on the row index value or the column index value, the rows and columns of the first matrix are reordered, Obtaining a second matrix so that the cache hit rate of the second matrix is ​​higher than that of the first matrix; Reading a third matrix from the memory of the GPU, and performing operations on the second matrix and the third matrix, wherein the third matrix refers to a matrix that operates on the second matrix, and the third matrix is ​​obtained based on the input sequence; The step of reordering the rows and columns of the first matrix based on the row index value or the column index value to obtain the second matrix includes: Taking the row index values ​​or column index values ​​of the first matrix as nodes, and performing node analysis on the first matrix to obtain the connection relationship between the nodes and the degree of the nodes, wherein the degree refers to the number of edges corresponding to the node; Reordering the rows and columns of the first matrix based on the connection relationship between the nodes and the degree of the nodes to obtain a second matrix; The reading of the third matrix from the memory of the GPU includes: Preprocessing is performed, and after the preprocessing, a repetitive operation is performed with a synchronization cycle as a minimum unit, Until the synchronization period satisfies: In the xth synchronization cycle, load the n-1th third matrix into the register memory; load the nth matrix block column index into the shared memory; load the n-1th second matrix into the shared memory, and calculate the n-2th matrix operation, wherein the n-2th matrix operation is obtained based on the n-2th second matrix and the n-2th third matrix, wherein the loading of the n-1th third matrix is ​​obtained based on the loading of the n-1th matrix column index in the x-2th synchronization cycle, x≥3, and the preprocessing is used to provide calculation data for subsequent synchronization cycles; The tensor processing unit includes a thread block, and the operation on the second matrix and the third matrix includes: Get the total processing time of the thread block processing the original task; reallocating the original tasks according to the total processing time of the thread blocks so that the distribution of the reallocated total processing time is more consistent with respect to the total processing time of the original tasks; The total processing time T of the thread block is calculated as: T=tcm*(lot+cot)+rwm*wt, Among them, tcm is the number of matrix blocks processed, lot is the preset single data acquisition time, cot is the preset single calculation time, rwm is the number of windows corresponding to the processed matrix blocks, and wt is the preset single write-back time.

2. The method for accelerating sparse matrix calculation on a tensor processing unit according to claim 1, characterized in that: The step of reordering the rows and columns of the first matrix based on the connection relationship between the nodes and the degrees of the nodes to obtain a second matrix includes: Merging the nodes based on the degree, and expanding the merged nodes to obtain a first array, wherein the nodes in the first array are arranged in a first order; Modifying the first array based on the connection relationship between the nodes to obtain a second array, wherein the nodes in the first array are arranged in a second order; Denodeization is performed based on the connection relationship between the second array and the nodes to obtain a second matrix.

3. The method for accelerating sparse matrix calculation on a tensor processing unit according to claim 2, characterized in that: The nodes are merged based on the size of the degree, and the merged nodes are expanded to obtain a first array, including: Sort the nodes according to the sizes of the degrees to obtain a third array, wherein the third array is a node set; Along the third array, calculating and obtaining the connection degree increment between adjacent nodes, wherein the connection degree increment represents the priority of the node merging order; Merging the nodes layer by layer according to the connectivity increment until the number of merged nodes is the same as the number of communities in the first matrix; Expand the merged nodes in a tree diagram to obtain a first array, wherein the first array refers to a node set formed along the leaf node direction of the tree diagram, and each layer of the tree diagram records the merging of nodes; or, The performing inverse nodeization based on the connection relationship between the second array and the nodes to obtain a second matrix includes: Mapping the second array to obtain a fourth array, wherein the fourth array is composed of node mappings, wherein the node mappings are obtained by mapping nodes in the second array based on a mapping rule; The fourth array is matrixed to obtain a third matrix.

4. The method for accelerating sparse matrix calculation on a tensor processing unit according to claim 2, characterized in that: The step of modifying the first array based on the connection relationship between the nodes to obtain the second array includes: Determine a node with a first priority, and determine another node closest to the node with the first priority in a second-order relationship based on the connection relationship between the nodes, to obtain a node with a second priority; Determine another node closest to the second-order relationship of the node at the 1+Nth order based on the connection relationship between the nodes, obtain the node at the 2+Nth order, and obtain the order of all nodes in the third array, where N is a positive integer; The nodes are arranged based on the sequence to obtain a second array.

5. The method for accelerating sparse matrix calculation on a tensor processing unit according to claim 4, characterized in that: When determining the nearest node in the second-order relationship, if the nearest node in the second-order relationship is not unique, the node ranked first in the first array is used as the nearest node in the second-order relationship, wherein the second-order relationship does not include nodes that have already obtained a sequence position.

6. The method for accelerating sparse matrix calculation on a tensor processing unit according to any one of claims 1 to 5, characterized in that: Before reading the third matrix from the memory of the GPU, the method further includes: The second matrix is ​​divided into M windows according to a preset first division rule, wherein the M windows are connected in sequence; The window is segmented into blocks based on non-zero elements to obtain a segmentation result consisting of matrix blocks; Based on the segmentation result, a window matrix block bias, a previous non-zero element bias of the matrix block, a matrix block column index, a non-zero element position set of the matrix block, and a non-zero element numerical sequence are obtained, wherein the matrix block column index refers to the index of the column of the second matrix where the non-zero elements in the matrix block are located, and the non-zero element numerical sequence is obtained based on the non-zero elements in the matrix block; the non-zero element position set of the matrix block is used to express the position arrangement of the non-zero elements of the matrix block itself.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program, when executed, executes the method for accelerating sparse matrix calculation on a tensor processing unit according to any one of claims 1 to 6.

Images