Data processing method and device for sparse matrix multiplication calculation, equipment and medium

Abstract

The application discloses a data processing method, device and equipment and medium for sparse matrix multiplication calculation. In the method, a sparse matrix is preprocessed to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; according to the number of non-zero elements in the sparse index, an index coordinate corresponding to a calculation unit PE in a one-dimensional systolic array is allocated; multiplication and accumulation calculation is performed according to a first calculation path in the PE to output a calculation result; multiplication calculation is performed according to a second calculation path in the PE to output a multiplication result corresponding to the PE; grouping summation is performed through a grouping summation network to obtain at least one grouping summation result; and the at least one grouping summation result is reordered to obtain an output matrix. In this way, the characteristics of matrix inner product and matrix row product are fully utilized to efficiently process SDDMM and SpMM, two types of sparse matrix multiplication, because the intermediate calculation results are temporarily stored and reused, the data movement and storage overhead are reduced, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device and medium for sparse matrix multiplication calculation. Background Art

[0002] The Transformer is a deep learning model based on the attention mechanism that has achieved remarkable results in fields such as natural language processing. In the field of deep learning, especially in calculations involving the Transformer model, matrix multiplication is one of the core operations.

[0003] However, as the model size increases, traditional dense matrix multiplication faces problems such as high computational resource consumption and long computation time. Sparse matrix multiplication, as an optimization method, reduces unnecessary computation by utilizing the large number of zero or negligible elements in the matrix, thereby improving data processing efficiency.

[0004] However, after a dense matrix is ​​made sparse, the non-zero elements therein are usually irregularly distributed, which makes it impossible for traditional processors to efficiently process sparse matrix data. In other words, the data processing efficiency of sparse matrix multiplication in the existing technology is low. Summary of the Invention

[0005] The present application provides a data processing method, apparatus, device and medium for sparse matrix multiplication calculation, which can improve the processing efficiency of irregular sparse matrices.

[0006] In a first aspect, the present application provides a data processing method for sparse matrix multiplication calculation, the method comprising:

[0007] Preprocessing the sparse matrix to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; the sparse index includes the number of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements;

[0008] assigning corresponding index coordinates to computing elements (PEs) in a one-dimensional systolic array according to the number of non-zero elements in the sparse index; the index coordinates comprising a row index and a column index; the one-dimensional systolic array comprising a predetermined number of PEs arranged in parallel; the PEs being reconfigurable PEs comprising a first computing path and a second computing path; the first computing path being configured to perform a matrix inner product operation, and the second computing path being configured to perform a matrix row product operation;

[0009] performing a multiplication-accumulation calculation on a corresponding row vector in a query matrix and a corresponding column vector in a transposed key matrix corresponding to a key matrix according to the first calculation path in the PE, and outputting a calculation result corresponding to the PE; the corresponding row vector in the query matrix corresponds to the row index of the PE, and the corresponding column vector in the transposed key matrix corresponding to the key matrix corresponds to the column index of the PE;

[0010] performing a multiplication operation on the calculation result and a corresponding row vector in a value matrix according to the second calculation path in the PE, and outputting a multiplication result corresponding to the PE; the corresponding row vector in the value matrix corresponds to the column index of the PE;

[0011] Performing group summation on the multiplication results having the number of non-zero elements through a group summation network to obtain at least one group summation result;

[0012] At least one of the grouped summation results is reordered according to a row index corresponding to the grouped summation result to obtain an output matrix; the output matrix corresponds to the sparse matrix block.

[0013] Optionally, the first computation path includes a first multiplexer, a second multiplexer, a third multiplexer, a fourth multiplexer, a multiplier, and an adder. The performing of multiplication-accumulation calculations on the corresponding row vectors in the query matrix and the corresponding column vectors in the transposed key matrix corresponding to the key matrix according to the first computation path in the PE, and outputting the calculation results corresponding to the PE, includes:

[0014] Traversing the row elements in the corresponding row vector of the query matrix and the column elements in the corresponding column vector of the transposed key matrix corresponding to the key matrix; the column index of the row element corresponds to the row index of the column element;

[0015] Inputting the row elements into the multiplier through the first multiplexer;

[0016] Input the column elements into the multiplier through the second multiplexer;

[0017] Performing a multiplication calculation on the row element and the row element according to the multiplier to obtain an element calculation result;

[0018] calling the third multiplexer to input the existing element calculation result and the sum into the adder; the existing element calculation result and the sum of the element calculation results that have been accumulated by the adder;

[0019] Accumulate the element calculation result and the existing element calculation results according to the adder to obtain the calculation result;

[0020] output the calculation result through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0021] Optionally, the second calculation path comprises a first multiplexer, a second multiplexer, a fourth multiplexer and a multiplier, and the multiplication calculation of the calculation result and the corresponding row vector in the value matrix is performed according to the second calculation path in the PE, and the multiplication result corresponding to the PE is output, comprising:

[0022] The calculation result is input into the multiplier through the first multiplexer.

[0023] The corresponding row vector in the value matrix is input into the multiplier through the second multiplexer.

[0024] The multiplication calculation of the calculation result and the corresponding row vector in the value matrix is performed according to the multiplier, and the multiplication result is obtained.

[0025] The multiplication result is output through the fourth multiplexer to complete the multiplication of the sparse matrix and the dense matrix SpMM.

[0026] Optionally, before the multiplication accumulation calculation of the corresponding row vector in the query matrix and the corresponding column vector in the corresponding transpose key matrix of the key matrix is performed according to the first calculation path in the PE, and the calculation result corresponding to the PE is output, further comprising:

[0027] Based on the row index of the PE, the corresponding row vector in the query matrix is input into the PE through the first cross network.

[0028] Based on the column index of the PE, the corresponding column vector in the corresponding transpose key matrix of the key matrix is input into the PE through the first cross network.

[0029] Optionally, before the multiplication calculation of the calculation result and the corresponding row vector in the value matrix is performed according to the second calculation path in the PE, and the multiplication result corresponding to the PE is output, further comprising:

[0030] Based on the column index of the PE, the corresponding row vector in the value matrix is input into the PE through the second cross network.

[0031] Optionally, the inputting of the corresponding row vector in the value matrix into the PE through the second cross network comprises:

[0032] According to the number of periods, the row elements in the corresponding row vector in the value matrix are input into the PE through the second cross network period by period, and the number of periods is equal to the number of columns of the value matrix.

[0033] Optionally, the pre-processing of the sparse matrix to obtain the sparse matrix block and the sparse index corresponding to the sparse matrix block comprises:

[0034] The sparse matrix is divided into blocks to obtain the sparse matrix block;

[0035] The number of non-zero elements and the index coordinates of the non-zero elements in the sparse matrix block are obtained;

[0036] The number of non-zero elements and the index coordinates are arranged in one dimension to obtain the sparse index corresponding to the sparse matrix block.

[0037] Optionally, the output of the calculation result by the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM comprises:

[0038] It is judged whether the row element is the last row element in the corresponding row vector in the query matrix;

[0039] If the row element is not the last row element in the corresponding row vector in the query matrix, the iteration of the row element in the corresponding row vector in the query matrix and the column element in the corresponding column vector in the transposed key matrix corresponding to the key matrix is performed until the last row element in the corresponding row vector in the query matrix is obtained;

[0040] If the row element is the last row element in the corresponding row vector in the query matrix, the calculation result is output by the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0041] Optionally, the output of the calculation result by the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM comprises:

[0042] It is judged whether the column element is the last column element in the corresponding column vector in the transposed key matrix corresponding to the key matrix;

[0043] If the column element is not the last column element in the corresponding column vector in the transposed key matrix corresponding to the key matrix, the iteration of the row element in the corresponding row vector in the query matrix and the column element in the corresponding column vector in the transposed key matrix corresponding to the key matrix is performed until the last column element in the corresponding column vector in the transposed key matrix corresponding to the key matrix is obtained;

[0044] If the column element is the last column element in the corresponding column vector in the transposed key matrix corresponding to the key matrix, the calculation result is output by the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0045] In a second aspect, the present application further provides a data processing device for sparse matrix multiplication calculation, the device comprising: a sparse index preprocessing module, a distribution network module, a one-dimensional systolic array module, and a group summation network module;

[0046] The sparse index preprocessing module is used to preprocess the sparse matrix to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; the sparse index includes the number of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements;

[0047] The distribution network module is configured to allocate corresponding index coordinates to computing elements (PEs) in a one-dimensional systolic array according to the number of non-zero elements in the sparse index; the index coordinates include a row index and a column index; the one-dimensional systolic array includes a preset number of PEs arranged in parallel; the PEs are reconfigurable PEs including a first computing path and a second computing path; the first computing path is configured to perform a matrix inner product operation, and the second computing path is configured to perform a matrix row product operation;

[0048] The one-dimensional systolic array module is configured to perform a multiplication-accumulation calculation on a corresponding row vector in a query matrix and a corresponding column vector in a transposed key matrix corresponding to a key matrix according to the first calculation path in the PE, and output a calculation result corresponding to the PE; the corresponding row vector in the query matrix corresponds to the row index of the PE, and the corresponding column vector in the transposed key matrix corresponding to the key matrix corresponds to the column index of the PE;

[0049] The one-dimensional systolic array module is further configured to perform a multiplication operation on the calculation result and a corresponding row vector in a value matrix according to the second calculation path in the PE, and output a multiplication result corresponding to the PE; the corresponding row vector in the value matrix corresponds to the column index of the PE;

[0050] The group summation network module is used to perform group summation on the multiplication results of the number of non-zero elements through a group summation network to obtain at least one group summation result;

[0051] The group summation network module is further used to reorder at least one of the group summation results according to the row index corresponding to the group summation result to obtain an output matrix; the output matrix corresponds to the sparse matrix block.

[0052] Optionally, the first computing path includes a first multiplexer, a second multiplexer, a third multiplexer, a fourth multiplexer, a multiplier, and an adder, and the one-dimensional systolic array module is specifically configured to:

[0053] Traversing the row elements in the corresponding row vector of the query matrix and the column elements in the corresponding column vector of the transposed key matrix corresponding to the key matrix; the column index of the row element corresponds to the row index of the column element;

[0054] Inputting the row elements into the multiplier through the first multiplexer;

[0055] Input the column elements into the multiplier through the second multiplexer;

[0056] Performing a multiplication calculation on the row element and the row element according to the multiplier to obtain an element calculation result;

[0057] calling the third multiplexer to input the existing element calculation result and the sum into the adder; the existing element calculation result and the sum of the element calculation results that have been accumulated by the adder;

[0058] Accumulate the element calculation result and the existing element calculation results according to the adder to obtain the calculation result;

[0059] The calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0060] Optionally, the second computing path includes a first multiplexer, a second multiplexer, a fourth multiplexer, and a multiplier, and the one-dimensional systolic array module is specifically configured to:

[0061] inputting the calculation result into the multiplier through the first multiplexer;

[0062] Inputting the corresponding row vector in the value matrix into the multiplier through the second multiplexer;

[0063] performing a multiplication calculation on the calculation result and a corresponding row vector in the value matrix according to the multiplier to obtain the multiplication result;

[0064] The multiplication result is output through the fourth multiplexer to complete the multiplication SpMM of the sparse matrix and the dense matrix.

[0065] Optionally, the one-dimensional systolic array module is specifically used to:

[0066] Determine whether the row element is the last row element in the corresponding row vector in the query matrix;

[0067] If the row element is not the last row element in the row vector corresponding to the query matrix, returning to execute the traversal of the row elements in the row vector corresponding to the query matrix and the column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix until the last row element in the row vector corresponding to the query matrix is ​​obtained;

[0068] If the row element is the last row element in the corresponding row vector in the query matrix, the calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0069] Optionally, the one-dimensional systolic array module is specifically used to:

[0070] Determine whether the column element is the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix;

[0071] If the column element is not the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix, then returning to execute the traversal of the row elements in the corresponding row vector in the query matrix and the column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix until the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix is ​​obtained;

[0072] If the column element is the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix, the calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0073] Optionally, the distribution network module is further configured to:

[0074] Based on the row index of the PE, inputting the corresponding row vector in the query matrix into the PE through the first cross network;

[0075] Based on the column index of the PE, a column vector corresponding to the transposed key matrix corresponding to the key matrix is ​​input into the PE through the first cross network.

[0076] Optionally, the distribution network module is further configured to:

[0077] Based on the column index of the PE, the corresponding row vector in the value matrix is ​​input into the PE through the second cross network.

[0078] Optionally, the distribution network module is specifically configured to:

[0079] The row elements in the corresponding row vectors in the value matrix are input into the PE periodically through the second cross network according to the number of cycles, and the number of cycles is equal to the number of columns in the value matrix.

[0080] The sparse index preprocessing module is specifically used to:

[0081] Dividing the sparse matrix into blocks to obtain the sparse matrix blocks;

[0082] Obtaining the number of non-zero elements and the index coordinates of the non-zero elements in the sparse matrix block;

[0083] The number of non-zero elements and the index coordinates are arranged in one dimension to obtain the sparse index corresponding to the sparse matrix block.

[0084] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory:

[0085] The memory is used to store computer programs;

[0086] The processor is configured to execute the method provided in the first aspect according to the computer program.

[0087] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method provided in the first aspect above.

[0088] It can be seen that this application has the following beneficial effects:

[0089] The present application provides a data processing method, apparatus, device and medium for sparse matrix multiplication calculation, in which the sparse matrix is ​​preprocessed to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; the sparse index includes the number of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements; according to the number of non-zero elements in the sparse index, the corresponding index coordinates are allocated to the computing unit PE in the one-dimensional systolic array; the index coordinates include row index and column index; the one-dimensional systolic array includes a preset number of PEs arranged in parallel; the PE is a reconfigurable PE including a first computing path and a second computing path; the first computing path is used to perform a matrix inner product operation, and the second computing path is used to perform a matrix row product operation; according to the first computing path in the PE, the query matrix The corresponding row vector and the corresponding column vector in the transposed key matrix corresponding to the key matrix are multiplied and accumulated, and the calculation result corresponding to PE is output; the corresponding row vector in the query matrix corresponds to the row index of PE, and the corresponding column vector in the transposed key matrix corresponding to the key matrix corresponds to the column index of PE; according to the second calculation path in PE, the calculation result and the row vector corresponding to the column index in the value matrix are multiplied, and the multiplication result corresponding to PE is output; the corresponding row vector in the value matrix corresponds to the column index of PE; the multiplication results of the number of non-zero elements are grouped and summed through the group summation network to obtain at least one group summation result; according to the row index corresponding to the group summation result, at least one group summation result is reordered to obtain the output matrix; the output matrix corresponds to the sparse matrix block.

[0090] Thus, the method provided in the embodiment of the present application, on the one hand, targets the irregular sparsity of sparse matrices by preprocessing the sparse matrices so that irregular elements can be tightly connected, effectively addressing the problem of irregular distribution of elements in the sparse matrix and improving the processing capability of irregular sparse matrices. At the same time, directly storing the non-zero elements in the sparse matrix blocks in the form of index coordinates not only simplifies the hardware logic for decoding the sparse format and improves data processing efficiency, but also can quickly obtain the index coordinates of the required elements, improve the efficiency of data access, and thus improve data processing efficiency. On the other hand, compared with the two-dimensional array of PEs in the prior art, the use of PEs arranged in parallel in a one-dimensional systolic array can reduce the idle rate of PEs during the calculation process, make full use of the parallelism and scalability of the hardware, improve the utilization rate of hardware resources, and improve data processing efficiency under limited hardware resources. On the other hand, the matrix inner product operation is realized through the first calculation path in the PE, and the matrix row product operation is realized through the second calculation path in the PE. The characteristics of the matrix inner product and the matrix row product are fully utilized to efficiently process the two types of sparse matrix multiplications SDDMM and SpMM. In addition, because the calculation results between SDDMM and SpMM are temporarily stored and reused, the overhead of storing and repeatedly moving the intermediate calculation results is saved, thereby improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0092] Figure 1 Schematic diagram of a flow chart of a data processing method for sparse matrix multiplication calculation in an embodiment of the present application;

[0093] Figure 2 A schematic diagram of sparse matrix preprocessing provided in an embodiment of the present application;

[0094] Figure 3 A schematic diagram of the overall flow of a data processing method for sparse matrix multiplication calculation provided in an embodiment of the present application;

[0095] Figure 4 A schematic diagram of inputting a one-dimensional systolic array through a distribution network provided in an embodiment of the present application;

[0096] Figure 5 A schematic diagram of the overall structure of a reconfigurable PE provided in an embodiment of the present application;

[0097] Figure 6 A schematic diagram of the structure of the first computing path of the reconfigurable PE provided in an embodiment of the present application;

[0098] Figure 7 A schematic diagram illustrating an example of a process for calculating a matrix inner product and a matrix outer product according to an embodiment of the present application;

[0099] Figure 8 A schematic diagram of the structure of the second computing path of the reconfigurable PE provided in an embodiment of the present application;

[0100] Figure 9 A schematic diagram of the group summation process provided in an embodiment of the present application;

[0101] Figure 10 A comparison chart of computational efficiency under different sparsity levels provided in the embodiments of this application;

[0102] Figure 11 A schematic structural diagram of a data processing device 1100 for sparse matrix multiplication calculation provided in an embodiment of the present application;

[0103] Figure 12 A schematic diagram of the overall process execution of a data processing device 1100 for sparse matrix multiplication calculation provided in an embodiment of the present application;

[0104] Figure 13This is a structural diagram of an electronic device 1300 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0105] The "multiple" involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first" and "second" are only used for the purpose of distinguishing the description and cannot be understood as indicating or implying relative importance or order.

[0106] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, the following further describes the embodiments of the present application in detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present application and are not intended to limit the present application. In addition, it should be noted that, for ease of description, the drawings only show portions relevant to the present application, not all structures.

[0107] As model size increases, traditional dense matrix multiplication faces challenges such as high computational resource consumption and long computation times. Sparse matrix multiplication, as an optimization technique, reduces unnecessary computation by utilizing the large number of zero or negligible elements in matrices, thereby improving data processing efficiency.

[0108] However, after a dense matrix is ​​made sparse, its non-zero elements are often irregularly distributed, which makes it impossible for traditional processors to efficiently process sparse matrix data. In other words, the data processing efficiency of sparse matrix multiplication in the existing technology is low.

[0109] The applicant's research found that traditional processors cannot efficiently process sparse matrix data for the following reasons:

[0110] ① For example, the computational complexity of the Transformer model is extremely high, especially the matrix multiplication operation in the self-attention mechanism, which uses a two-dimensional PE computing unit. The calculation process of the two-dimensional PE computing unit occupies a large amount of computing resources and time, and cannot fully utilize the performance of the hardware, resulting in low data processing efficiency.

[0111] ②Under the trend of sparsification of the Transformer model, sparse matrix multiplication becomes the key to improving computational efficiency. However, existing sparse matrix processing methods have many problems. On the one hand, irregular sparse methods affect hardware execution efficiency; on the other hand, regular sparse methods often require retraining of the model, resulting in additional training overhead. Among them, in the irregular sparse method, the number of non-zero elements in each row and each column of the sparse matrix is not fixed. The distribution of matrix elements in this method is uneven, and some rows or columns may contain more non-zero elements; among them, in the regular sparse method, the number of non-zero elements in each row and each column of the sparse matrix is fixed. This means that the distribution of elements in the matrix is uniform, and the frequency of occurrence of each element is roughly the same.

[0112] ③In the process of deep learning calculation, the data sources and forms are diverse, and the data generated at different stages have different characteristics and requirements. For example, in the sparse attention mechanism, there are various data streams such as full-precision data, quantized data, and data processed by pruning and sorting. Existing technologies cannot well integrate these complex data streams, resulting in low data transmission efficiency, waste of computing resources, and low data processing efficiency.

[0113] In summary, in order to solve the problems of low data processing efficiency caused by low hardware resource utilization, low adaptability of sparse matrices, and poor integration of complex data streams corresponding to the Transformer model deep learning model, the embodiments of the present application provide a data processing method, device, equipment and medium for sparse matrix multiplication calculation. In this method, the sparse matrix is ​​preprocessed to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; the sparse index includes the number of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements; according to the number of non-zero elements in the sparse index, the corresponding index coordinates are assigned to the computing unit PE in the one-dimensional systolic array; the index coordinates include row index and column index; the one-dimensional systolic array includes a preset number of PEs arranged in parallel; the PE is a reconfigurable PE including a first computing path and a second computing path; the first computing path is used to perform matrix inner product Operation, the second calculation path is used to perform matrix row product operation; according to the first calculation path in PE, the corresponding row vector in the query matrix and the corresponding column vector in the transposed key matrix corresponding to the key matrix are subjected to multiplication and accumulation calculation, and the calculation result corresponding to PE is output; the corresponding row vector in the query matrix corresponds to the row index of PE, and the corresponding column vector in the transposed key matrix corresponding to the key matrix corresponds to the column index of PE; according to the second calculation path in PE, the multiplication calculation is performed on the calculation result and the row vector corresponding to the column index in the value matrix, and the multiplication result corresponding to PE is output; the corresponding row vector in the value matrix corresponds to the column index of PE; the multiplication results of the number of non-zero elements are grouped and summed through the group summation network to obtain at least one group summation result; according to the row index corresponding to the group summation result, at least one group summation result is reordered to obtain the output matrix; the output matrix corresponds to the sparse matrix block.

[0114] Thus, the method provided by the embodiment of the present application, on the one hand, targets the irregular sparsity of sparse matrices, and by pre-processing the sparse matrix, the irregular elements can be tightly connected, effectively coping with the problem of irregular distribution of elements in the sparse matrix, and improving the processing capability of irregular sparse matrices. At the same time, directly storing the non-zero elements in the sparse matrix block in the form of index coordinates not only simplifies the hardware logic of decoding the sparse format and improves data processing efficiency, but also can quickly obtain the index coordinates of the required elements, improves the efficiency of data access, and thus improves data processing efficiency. On the other hand, the matrix inner product operation is realized by the first calculation path in the PE, and the matrix row product operation is realized by the second calculation path in the PE, making full use of the characteristics of the matrix inner product and the matrix row product to efficiently process the two types of sparse matrix multiplications, SDDMM and SpMM, and realizing the sparse matrix multiplication calculation that integrates multiple data streams. Moreover, because the calculation results between SDDMM and SpMM are temporarily stored and reused, the overhead of storing and repeatedly transporting the intermediate calculation results is saved, thereby improving data processing efficiency. On the other hand, compared with the two-dimensionally arranged PEs in the prior art, the embodiment of the present application uses computing units PEs arranged in parallel in a one-dimensional systolic array for calculation, fully utilizing the parallelism and scalability of the hardware, which can improve the utilization of hardware resources and improve data processing efficiency under limited hardware resources.

[0115] To facilitate understanding of the specific implementation of the data processing method for sparse matrix multiplication calculation provided in the embodiment of the present application, it will be explained below with reference to the accompanying drawings.

[0116] It should be noted that the subject implementing the data processing method for sparse matrix multiplication calculation can be the data processing device for sparse matrix multiplication calculation provided in the embodiment of the present application, and the data processing device for sparse matrix multiplication calculation can be carried in an electronic device or a functional module of an electronic device. The electronic device in the embodiment of the present application can be any device that can implement the data processing method for sparse matrix multiplication calculation in the embodiment of the present application, for example, it can be an Internet of Things (IoT) device.

[0117] See also Figure 1 , provides a flow chart of a data processing method for sparse matrix multiplication calculation. In an embodiment of the present application, for example, the following steps may be included:

[0118] S1, preprocessing the sparse matrix to obtain sparse matrix blocks and sparse indices corresponding to the sparse matrix blocks.

[0119] It should be noted that the sparse index includes the number of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements. A sparse matrix refers to a data set in which most elements in the two-dimensional table are zero and there are relatively few non-zero elements.

[0120] In a possible implementation, S1 provided in the embodiment of the present application may include:

[0121] S101, dividing the sparse matrix into blocks to obtain sparse matrix blocks.

[0122] S102, obtaining the number of non-zero elements and the index coordinates of the non-zero elements in the sparse matrix block.

[0123] S103 , performing one-dimensional arrangement on the number of non-zero elements and the index coordinates to obtain a sparse index corresponding to the sparse matrix block.

[0124] It should be noted that the specific execution process of steps S101 to S103 can be found in Figure 2 , Figure 2 A schematic diagram of sparse matrix preprocessing is provided.

[0125] exist Figure 2 In the sparse matrix preprocessing process shown, first, the sparse matrix is ​​divided into blocks, the sparse matrix is ​​an 8×8 matrix, and the sparse matrix is ​​divided into 6 sparse matrix blocks; then, for each sparse matrix block, the number of non-zero elements and the index coordinates of the non-zero elements in the sparse matrix block are obtained, and the number of non-zero elements and the index coordinates are arranged in one dimension to obtain the sparse index of the sparse matrix block, wherein the white squares in the sparse matrix represent zero elements, and the non-white squares in the sparse matrix represent non-zero elements. The generated index is stored in the on-chip cache to increase the speed of acquiring data. In the embodiment of the present application, the number of non-zero elements in the sparse matrix block A is 4, and the index coordinates of the non-zero elements are (0, 0), (0, 2), (1, 1) and (2, 3), respectively, that is, the sparse index includes the index coordinates (0, 0), (0, 2), (1, 1), (2, 3) and the number of non-zero elements 4.

[0126] In a specific implementation, the sparse matrix can be divided into blocks based on the number of PEs in the one-dimensional systolic array (see the one-dimensional systolic array in step S2 for details. The one-dimensional systolic array provided in this embodiment of the present application includes four PEs). The number of PEs in the one-dimensional systolic array is equal to the number of columns in the sparse matrix block. The coordinates of non-zero elements are read in one by one and judged. Then, they are placed in the corresponding block through a first-in-first-out queue. The embodiment of the present application does not limit the number of rows in the sparse matrix block during block division. That is, the six sparse matrix blocks of the sparse matrix are, from top to bottom and from left to right, 3×4 matrix, 3×4 matrix, 3×4 matrix, 2×4 matrix, and 2×4 matrix.

[0127] In an embodiment of the present application, by dividing the sparse matrix into blocks and arranging it in one dimension, irregular data can be tightly connected to facilitate subsequent calculations. This preprocessing method can effectively deal with the problem of irregular distribution of elements in the sparse matrix and improve the processing capability of irregular sparse matrices.

[0128] Furthermore, in the embodiments of the present application, the COO format is used to directly store the non-zero elements in the sparse matrix block as index coordinates, reducing the amount of storage required and enabling rapid access to the index coordinates of the required elements during calculations. This improves data access efficiency and, in turn, data processing efficiency. Directly storing coordinates eliminates the need for decoding, simplifying the hardware logic for decoding the sparse format and, to a certain extent, also improving data processing efficiency.

[0129] S2, assigns corresponding index coordinates to the computing units PE in the one-dimensional systolic array according to the number of non-zero elements in the sparse index.

[0130] It should be noted that the index coordinates include row indices and column indices; the one-dimensional systolic array includes a preset number of PEs arranged in parallel, and the PE is a reconfigurable PE including a first computing path and a second computing path. The first computing path is used to perform matrix inner product operations, and the second computing path is used to perform matrix row product operations.

[0131] In the embodiments of the present application, a one-dimensional systolic array includes a preset number of PEs arranged in parallel. Reconfigurable PEs mean that each PE can be configured to calculate both matrix inner products and matrix row products. The preset number can be set as needed and is not limited by the embodiments. To facilitate understanding of the embodiments of the present application, a preset number of four, i.e., a one-dimensional systolic array including four parallel PEs, is used as an example for description.

[0132] In one possible implementation, please refer to Figure 3 , Figure 3 A schematic diagram of the overall flow of a data processing method for sparse matrix multiplication calculation provided in an embodiment of the present application.

[0133] In the embodiment of the present application, the corresponding index coordinates are allocated to the PEs in the one-dimensional systolic array according to the number of non-zero elements in the sparse index. In the above example, the sparse index corresponding to the sparse matrix block A includes index coordinates (0, 0), (0, 2), (1, 1), (2, 3) and the number of non-zero elements is 4. Then, the index coordinates of the non-zero elements to be processed are allocated to the four parallel PEs respectively, that is, PE0 corresponds to the index coordinate (0, 0), PE1 corresponds to the index coordinate (0, 2), PE2 corresponds to the index coordinate (1, 1), and PE3 corresponds to the index coordinate (2, 3), where c represents the column number variable.

[0134] It should be noted that in the above example, the number of non-zero elements in the sparse matrix block is exactly 4, which is exactly equal to the number of PEs arranged in parallel in the one-dimensional systolic array. If the number of non-zero elements in a sparse matrix block B is 5, and the coordinates are (0, 1), (1, 2), (2, 1), (2, 3) and (3, 2) respectively, then in the process of assigning index coordinates to PEs in the one-dimensional systolic array, the four PEs should first be assigned index coordinates of (0, 1), (1, 2), (2, 1), and (2, 3). After the PEs complete the corresponding calculations, the index coordinate (3, 2) is assigned to one of the PEs. Usually, the assignment is made in sequence, that is, the index coordinate (3, 2) is assigned to PE0.

[0135] Compared to the prior art method of arranging PEs in two dimensions, the one-dimensional systolic array provided in the embodiments of the present application arranges PEs in parallel and assigns sparse coordinates to the PEs in the one-dimensional systolic array based on the number of non-zero elements in the sparse matrix block. This reduces the number of idle PEs in the computation process, improves computational efficiency, and reduces computational cost.

[0136] S3, according to the first calculation path in PE, perform multiplication and accumulation calculations on the corresponding row vector in the query matrix and the corresponding column vector in the transposed key matrix corresponding to the key matrix, and output the calculation result corresponding to PE.

[0137] Among them, the corresponding row vector in the query matrix corresponds to the row index of PE, and the corresponding column vector in the transposed key matrix corresponding to the key matrix corresponds to the column index of PE.

[0138] It should be noted that the query matrix is ​​usually represented by Q, and the key matrix is ​​usually represented by K. Through the first calculation path of the PE, the corresponding row vector in the query matrix Q input to the PE and the transposed matrix K corresponding to the key matrix K are calculated. T Perform multiplication and accumulation calculations on the corresponding column vectors in and output the calculation results corresponding to the PE.

[0139] In a possible implementation, before S3, the embodiment of the present application may further include:

[0140] Based on the row index of PE, the corresponding row vector in the query matrix is ​​input into PE through the first cross network;

[0141] Based on the column index of PE, the corresponding column vector in the transposed key matrix corresponding to the key matrix is ​​input into PE through the first cross network.

[0142] For specific implementation process, please refer to Figure 4 A schematic diagram of a one-dimensional systolic array input through a distribution network is shown. In the embodiment of the present application, based on the index coordinates corresponding to PE, the corresponding row vector in the query matrix Q and the transposed matrix K corresponding to the key matrix K are converted intoT The corresponding column vector in is input into the corresponding PE through the first cross network. Among them, the query matrix Q and the key matrix K are both dense matrices, the query matrix Q is the left matrix, the key matrix K is the right matrix, and the PE is a reconfigurable PE.

[0143] In one possible implementation, the first computing path includes a first multiplexer, a second multiplexer, a third multiplexer, a fourth multiplexer, a multiplier, and an adder. For the specific structure of PE, see Figure 5 , Figure 5 A schematic diagram of the overall structure of the reconfigurable PE provided in an embodiment of the present application, wherein the first multiplexer is represented by M1, the second multiplexer is represented by M2, the third multiplexer is represented by M3, and the fourth multiplexer is represented by M4, that is, the first computing path includes: the first multiplexer M1, the second multiplexer M2, the third multiplexer M3, the fourth multiplexer M4, a multiplier and an adder, and 16 or 32 represents the data bit width, that is, 16 bits or 32 bits.

[0144] The specific execution process of S3 provided in this embodiment of the application can be found in Figure 6 and Figure 7 , Figure 6 This is a schematic diagram of the structure of the first computing path of the reconfigurable PE provided in an embodiment of the present application. Figure 7 This is a schematic diagram of an example of a process for calculating a matrix inner product and a matrix outer product provided in an embodiment of the present application. S3 may include:

[0145] S31, traverse the row elements in the corresponding row vector in the query matrix and the column elements in the corresponding column vector in the transposed key matrix corresponding to the key matrix.

[0146] The column indices of the row elements correspond to the row indices of the column elements.

[0147] S32, input the row elements into the multiplier through the first multiplexer.

[0148] S33, input the column elements into the multiplier through the second multiplexer.

[0149] S34, performing multiplication calculation on the row elements and the row elements according to the multiplier to obtain the element calculation result.

[0150] S35, calling the third multiplexer to calculate the result of the existing elements and input it into the adder.

[0151] The sum of the existing element calculation results is the sum of the element calculation results that have been accumulated according to the adder.

[0152] S36, performing cumulative calculation on the element calculation result and the existing element calculation results according to the adder to obtain a calculation result;

[0153] S37 , outputting the calculation result through the fourth multiplexer to complete sampled dense-dense matrix multiplication (SDDMM).

[0154] Among them, SDDMM refers to the multiplication of some rows and columns of a dense matrix to obtain a sparse matrix.

[0155] Taking PE0 assigned index coordinate (0,0) as an example, it can be seen that the row index of PE0 is 0 and the column index is 0. Then, the row vector q0 corresponding to the first row in the query matrix Q is input into PE0 through the first distribution network, and the transposed matrix K corresponding to the key matrix K is T The column vector k0 corresponding to the first column in is input into PE0 through the first distribution network.

[0156] The multiplication and accumulation calculation is performed on q0 and k0 by the first calculation matrix in PE0, and the calculation result is 1.5, and the calculation result 1.5 corresponding to PE0 is output. In the specific execution process, the row elements in the row vector q0 corresponding to the first row in the query matrix Q are input into the multiplier one by one through M1, and the transposed matrix K corresponding to the key matrix K is output. T The column elements in the column vector k0 corresponding to the first column are input into the multiplier one by one through M2. The element calculation result corresponding to the first row element and the first column element obtained by the multiplier is 1.2×0.8=0.96. Then the element calculation result corresponding to the second row element and the second column element is 0.5×0.4=0.2. The third multiplexer M3 is called, and the existing element calculation results and 0.96 are input into the adder. The element calculation result 0.2 is accumulated with the existing element calculation results and 0.96 according to the adder... That is, 1.2×0.8+0.5×0.4+0.3×0.5+0.1×0.2+0.6×0.3+0.2×0.1=1.56. For the convenience of subsequent calculations, one decimal place is retained here. The final calculation result s is 1.5, and the calculation result s corresponding to PE0 is output as 1.5 through the fourth multiplexer.

[0157] Similarly, the calculation result s outputted by PE1 is 1.1, the calculation result s outputted by PE2 is 0.5, and the calculation result s outputted by PE3 is 0.5. In the embodiment of the present application, the above calculation result s is temporarily stored inside the PE so that it can be reused in the subsequent step S4 when performing the matrix row product calculation, saving the overhead of storing and repeatedly moving intermediate results.

[0158] It should be noted that the calculation result s corresponding to PE0 is 1.5, the calculation result s corresponding to PE1 is 1.1, the calculation result s corresponding to PE2 is 0.5, and the calculation result s corresponding to PE3 is 0.5, which together constitute the calculation result matrix S. For details, see Figure 5 The calculation result matrix S should be noted that this figure is for understanding that the calculation result s in the calculation result matrix S corresponds to the non-zero elements in the sparse matrix block, that is, the calculation result matrix S is also a sparse matrix.

[0159] In a specific implementation process, the embodiment of the present application provides outputting the calculation result through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM, including:

[0160] Determine whether the row element is the last row element in the corresponding row vector in the query matrix;

[0161] If the row element is not the last row element in the row vector corresponding to the query matrix, return to step S31, that is, return to traverse the row elements in the row vector corresponding to the query matrix and the column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix until the last row element in the row vector corresponding to the query matrix is ​​obtained;

[0162] If the row element is the last row element in the corresponding row vector in the query matrix, the calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0163] In a specific implementation process, the embodiment of the present application provides outputting the calculation result through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM, including:

[0164] Determine whether the column element is the last column element in the corresponding column vector in the transposed key matrix corresponding to the key matrix;

[0165] If the column element is not the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix, then return to execute the traversal of the row elements in the corresponding row vector in the query matrix and the column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix until the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix is ​​obtained;

[0166] If the column element is the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix, the calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0167] It should be noted that two judgment methods are given above to determine whether the calculation of all row elements in the row vector corresponding to the query matrix and all column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix has been completed. Because the number of row elements in the row vector corresponding to the query matrix is ​​equal to the number of column vectors corresponding to the transposed key matrix corresponding to the key matrix, one of the judgment methods can be selected for execution in the specific implementation process.

[0168] In an embodiment of the present application, in the SDDMM mode, the PEs in the one-dimensional systolic array are configured to perform multiplication and accumulation operations. Each row element of the left matrix and each row element of the right matrix (because each column element corresponding to the transposed matrix of the right matrix can be regarded as each row element of the right matrix) continuously flow into the corresponding PE for multiplication and accumulation, and are calculated in the form of matrix inner product to obtain the calculation result s, thereby completing the SDDMM. In this mode, the multiplexers M1 and M2 in the PE respectively transmit element q and element k into the multiplier for multiplication. The multiplexer M3 sends the partial sum to the adder and accumulates it with the new multiplication result until the element calculation in the PE is completed.

[0169] S4, according to the second calculation path in the PE, perform multiplication calculation on the calculation result and the corresponding row vector in the value matrix, and output the multiplication result corresponding to the PE.

[0170] Among them, the corresponding row vector in the value matrix corresponds to the column index of the PE, the calculation result is a part of the calculation result matrix, and the calculation result corresponding to each PE is temporarily stored in the PE.

[0171] It should be noted that the value matrix is ​​usually represented by V. Through the second calculation path of the PE, multiplication calculation is performed on the corresponding row vector in the value matrix V input to the PE, and the multiplication result corresponding to the PE is output.

[0172] In a possible implementation, before S4, the embodiment of the present application may further include:

[0173] Based on the column index of PE, the corresponding row vector in the value matrix is ​​input into PE through the second cross network.

[0174] In the specific implementation process, the value matrix V is a dense matrix, and the value matrix V is a right matrix. Please refer to Figure 4 A schematic diagram of inputting a one-dimensional systolic array through a distribution network is shown. In this embodiment of the present application, based on the column index corresponding to the PE, the corresponding row vector in the value matrix V is input to the PE through the second cross-connect network. Taking PE0 assigned index coordinates (0, 0) as an example, the column index of PE0 is 0. Then, the row vector V0 corresponding to the first row in the value matrix V is input to PE0 through the second distribution network.

[0175] In a possible implementation, the second computing path includes a first multiplexer, a second multiplexer, a fourth multiplexer, and a multiplier.

[0176] The specific execution process of S4 provided in this embodiment of the application can be found in Figure 7 and Figure 8 , Figure 7 A schematic diagram of an example of a calculation process of a matrix inner product and a matrix outer product provided in an embodiment of the present application, Figure 8 A schematic diagram of the structure of the second computing path of the reconfigurable PE provided in an embodiment of the present application, S4 may include:

[0177] S41, inputting the calculation result into the multiplier through the first multiplexer.

[0178] S42, inputting the corresponding row vector in the value matrix into the multiplier through the second multiplexer.

[0179] S43, performing a multiplication calculation on the calculation result and the corresponding row vector in the value matrix according to the multiplier to obtain a multiplication result.

[0180] S44 , outputting the multiplication result through the fourth multiplexer to complete sparse-dense matrix multiplication (SpMM).

[0181] Among them, SpMM refers to multiplying a sparse matrix by a dense matrix to obtain a dense matrix.

[0182] Still taking PE0 assigned index coordinate (0, 0) as an example, the calculation result s of 1.5 is temporarily stored in PE0. The calculation result s1.5 is input into the multiplier through the first multiplexer M1. The column index of PE0 is 0. Then, the row vector v0 corresponding to the first row in the value matrix V is input into the multiplier through the second multiplexer M2.

[0183] The multiplier performs multiplication on the calculation result s and the row vector v0 of the first row corresponding to the value matrix V to obtain the multiplication result. For PE0, calculate 1.5×v0 and obtain the multiplication result with z 0p0 Indicates that z 0p0 =[1.05, 0.75, 0.45, 1.35, 0.60, 0.90], and outputs the z value through the fourth multiplexer M4. 0 p 0 .

[0184] Similarly, output the multiplication result z corresponding to PE1 0p2 =[1.65, 2.53, 0.88, 1.32, 2.09, 055], output the multiplication result z corresponding to PE21p1 For z 1p1 =[0.25, 0.40, 0.15, 0.45, 0.20, 0.30], output the multiplication result z corresponding to PE3 2p3 =[0.65, 1.05, 0.45, 0.80, 0.55, 0.35].

[0185] In an embodiment of the present application, in SpMM mode, the PEs in a one-dimensional systolic array are configured to perform multiplication operations. The sparse elements in the left matrix S are temporarily stored in the PEs, and the elements of each row of the right matrix continuously flow into the corresponding PE for multiplication. The multiplication results are then fed back into the group summation network, where they are calculated in the form of matrix row products to obtain the multiplication result z, thereby completing SpMM. In this mode, multiplexers M1 and M2 in the PEs respectively transmit s and v to the multiplier for multiplication, and multiplexer M4 directly outputs the multiplication results without accumulating them within the PE.

[0186] In a possible implementation, the row elements in the corresponding row vectors in the value matrix may be input into the PE periodically through the second cross-network according to the number of cycles, where the number of cycles is equal to the number of columns in the value matrix.

[0187] To facilitate understanding of the solution provided by the embodiment of the present application, the following is explained with reference to PE0 as an example. The number of columns of the value matrix V provided by the embodiment of the present application is 6, so the number of cycles T = 6. In the first cycle, the first row element 0.7 in the row vector v0 corresponding to the first row of the value matrix V is input into PE0 through the second cross network. In the second cycle, the second row element 0.5 in the row vector v0 corresponding to the first row of the value matrix V is input into PE0 through the second cross network... In the sixth cycle, the sixth row element 0.6 in the row vector v0 corresponding to the first row of the value matrix V is input into PE0 through the second cross network, thereby completing the input of the corresponding row vector in the value matrix V through the second cross network.

[0188] According to the above steps S3 to S4, the computing scenario of the attention mechanism of the Transfomer model in the embodiment of the present application can convert the original dense matrix multiplication into SDDMM and SpMM, and then the intermediate results of SDDMM and SpMM can be temporarily stored and reused, saving the overhead of storing and repeatedly moving the intermediate results, thereby improving data processing efficiency.

[0189] The data processing method for sparse matrix multiplication calculation provided in the embodiment of the present application is oriented to the sparse attention mechanism of the Tansformer model, and supports the matrix multiplication structure of the matrix inner product (inner product) and the matrix row product (row-wise product). The embodiment of the present application can not only process SDDMM and SpMM, but is also applicable to general matrix matrix multiplication (GEneral Matrix to Matrix Multiplication, GEMM), and has high scalability and versatility.

[0190] In other words, the embodiments of the present application can support three matrix calculation modes: SDDMM, SpMM, and GEMM, and have high scalability and versatility. Furthermore, when expanding from a single computing structure to multiple ones, computing efficiency and resource usage scale linearly, with low expansion cost, which makes it possible to process large-scale data and more complex computing tasks.

[0191] S5, performing group summation on the multiplication results of the number of non-zero elements through a group summation network to obtain at least one group summation result.

[0192] In an embodiment of the present application, it is necessary to group and sum the multiplication results of non-zero elements through a group summation network to obtain at least one group summation result. In the specific implementation process, the multiplication results corresponding to PEs with the same row index are summed.

[0193] For the specific implementation process, please refer to Figure 9 Schematic diagram of the group summation process shown and Figure 7 The diagram of an example of the calculation process of the matrix inner product and the matrix outer product is shown in FIG. 1 , and the above example is used to illustrate that the non-zero element multiplication results, that is, the 4 multiplication results, are z 0p0 =[1.05, 0.75, 0.45, 1.35, 0.60, 0.90], z 0p2 =[1.65, 2.53, 0.88, 1.32, 2.09, 055], z 1p1 =[0.25, 0.40, 0.15, 0.45, 0.20, 0.30] and z 2p3 =[0.65, 1.05, 0.45, 0.80, 0.55, 0.35]. z 0p0 The corresponding row index of PE0 is 0, z 0p2 The corresponding PE1 row index is 0, for z 0p0 and z 0p2 Sum and finally get z 0p0 +z 0p2 =z 0p , z 0p=[2.70,3.28,1.33,2.67,2.69,1.45]. It should be noted that, because the other PE row indices are different, at least one group summation result is finally obtained, including z 0p 、z 1p and z 2p .

[0194] In the specific implementation process, the group summation network includes registers. Although the registers exist in the form of a tree topology, the number of registers at each level in the tree topology is the same as the preset number of PEs in the one-dimensional systolic array, thereby completing the discrimination and summation of elements in different groups.

[0195] It should be noted that the group summation network is designed to support the calculation mode of matrix row product. The matrix row product requires that the non-zero elements of a row of the calculation result matrix S (left matrix) are multiplied one by one with the elements in the corresponding row vector of the value matrix V (right matrix). The multiple rows obtained are then accumulated at the corresponding positions. Therefore, the group summation network determines whether it is a type of row that needs to be summed based on the row index saved in the PE. After that, each result of the same type of row is accumulated in the group summation network after being generated in the PE, and is no longer accumulated in the PE, thereby completing the summation of the intermediate result rows of the matrix row product.

[0196] In this embodiment, the group summation network performs group summation based on the row index within the PE, completing the summation of the intermediate results of the matrix row product. The group summation network can accurately sum the elements in the same row, avoiding data confusion and miscalculation, and improving the accuracy and efficiency of the calculation.

[0197] S6. Reorder at least one group summation result according to the row index corresponding to the group summation result to obtain an output matrix.

[0198] Among them, the output matrix corresponds to the sparse matrix block.

[0199] In the embodiment of the present application, since at least one group summation result obtained in step S5 includes z 0p 、z 1p and z 2p , at least one group summation result is not necessarily arranged in the order of arrangement, so it needs to be reordered to finally obtain the output matrix Z, which is the result obtained after the corresponding data processing of the sparse matrix block. The reordering makes the row index the same as the natural sequence number of the array.

[0200] In the specific implementation process, due to the step-by-step storage characteristics of the group summation network, each sum of its final output is distributed at any position where the corresponding group element is located, so it is also necessary to reorder each sum according to the row index so that its row index is the same as the natural sequence number of the array.

[0201] It should be noted that through the above series of designs, the accelerator can maintain a high hardware utilization rate under different sparsity. When the sparsity is high, the hardware efficiency of the embodiment of the present application is higher than other methods in the prior art. Because the computing unit PE can be fully utilized, the idle time is reduced and the output processing efficiency of the entire accelerator is improved. For specific comparison results, please refer to Figure 10 , Figure 10 A comparison chart of computing efficiency under different sparsities provided for an embodiment of the present application shows that the embodiment of the present application can still maintain high utilization and computing efficiency at higher sparsity, and has a 20% efficiency improvement compared to the traditional two-dimensional systolic array solution.

[0202] Thus, the method provided in the embodiment of the present application can uniformly support the calculation of two sparse modes, SDDMM and SpMM, and make full use of the characteristics of matrix inner product and matrix row product to efficiently process the two types of sparse matrix multiplication, SDDMM and SpMM. In the Transformer-oriented attention mechanism calculation scenario, Q×K T Corresponding to SDDMM, S×V corresponds to SpMM. By connecting the two, the calculation result s in the intermediate calculation result matrix S can be temporarily stored in the corresponding PE. After the calculation of both modes is completed, the calculation result s in PE can be directly discarded, reducing the storage and data exchange overhead.

[0203] See also Figure 11 , an embodiment of the present application further provides a data processing device 1100 for sparse matrix multiplication calculation, the device 1100 comprising: a sparse index preprocessing module 1101, a distribution network module 1102, a one-dimensional systolic array module 1103 and a group summation network module 1104;

[0204] The sparse index preprocessing module 1101 is used to preprocess the sparse matrix to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; the sparse index includes the number of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements;

[0205] The distribution network module 1102 is configured to allocate corresponding index coordinates to computing elements (PEs) in a one-dimensional systolic array according to the number of non-zero elements in the sparse index; the index coordinates include a row index and a column index; the one-dimensional systolic array includes a preset number of PEs arranged in parallel; the PEs are reconfigurable PEs including a first computing path and a second computing path; the first computing path is configured to perform a matrix inner product operation, and the second computing path is configured to perform a matrix row product operation;

[0206] The one-dimensional systolic array module 1103 is configured to perform a multiplication-accumulation calculation on a corresponding row vector in a query matrix and a corresponding column vector in a transposed key matrix corresponding to a key matrix according to the first computation path in the PE, and output a computation result corresponding to the PE; the corresponding row vector in the query matrix corresponds to the row index of the PE, and the corresponding column vector in the transposed key matrix corresponding to the key matrix corresponds to the column index of the PE;

[0207] The one-dimensional systolic array module 1103 is further configured to perform a multiplication operation on the calculation result and a corresponding row vector in a value matrix according to the second calculation path in the PE, and output a multiplication result corresponding to the PE; the corresponding row vector in the value matrix corresponds to the column index of the PE;

[0208] The group summation network module 1104 is configured to perform group summation on the multiplication results having the number of non-zero elements through a group summation network to obtain at least one group summation result;

[0209] The group summation network module 1104 is further configured to reorder at least one of the group summation results according to a row index corresponding to the group summation result to obtain an output matrix; the output matrix corresponds to the sparse matrix block.

[0210] Optionally, the first computing path includes a first multiplexer, a second multiplexer, a third multiplexer, a fourth multiplexer, a multiplier, and an adder, and the one-dimensional systolic array module 1103 is specifically configured to:

[0211] Traversing the row elements in the corresponding row vector of the query matrix and the column elements in the corresponding column vector of the transposed key matrix corresponding to the key matrix; the column index of the row element corresponds to the row index of the column element;

[0212] Inputting the row elements into the multiplier through the first multiplexer;

[0213] Input the column elements into the multiplier through the second multiplexer;

[0214] Performing a multiplication calculation on the row element and the row element according to the multiplier to obtain an element calculation result;

[0215] calling the third multiplexer to input the existing element calculation result and the sum into the adder; the existing element calculation result and the sum of the element calculation results that have been accumulated by the adder;

[0216] Accumulate the element calculation result and the existing element calculation results according to the adder to obtain the calculation result;

[0217] The calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0218] Optionally, the second computation path includes a first multiplexer, a second multiplexer, a fourth multiplexer, and a multiplier, and the one-dimensional systolic array module 1103 is specifically configured to:

[0219] inputting the calculation result into the multiplier through the first multiplexer;

[0220] Inputting the corresponding row vector in the value matrix into the multiplier through the second multiplexer;

[0221] performing a multiplication calculation on the calculation result and a corresponding row vector in the value matrix according to the multiplier to obtain the multiplication result;

[0222] The multiplication result is output through the fourth multiplexer to complete the multiplication SpMM of the sparse matrix and the dense matrix.

[0223] Optionally, the one-dimensional systolic array module 1103 is specifically configured to:

[0224] Determine whether the row element is the last row element in the corresponding row vector in the query matrix;

[0225] If the row element is not the last row element in the row vector corresponding to the query matrix, returning to execute the traversal of the row elements in the row vector corresponding to the query matrix and the column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix until the last row element in the row vector corresponding to the query matrix is ​​obtained;

[0226] If the row element is the last row element in the corresponding row vector in the query matrix, the calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0227] Optionally, the one-dimensional systolic array module 1103 is specifically configured to:

[0228] Determine whether the column element is the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix;

[0229] If the column element is not the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix, then returning to execute the traversal of the row elements in the corresponding row vector in the query matrix and the column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix until the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix is ​​obtained;

[0230] If the column element is the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix, the calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0231] Optionally, the distribution network module 1102 is further configured to:

[0232] Based on the row index of the PE, inputting the corresponding row vector in the query matrix into the PE through the first cross network;

[0233] Based on the column index of the PE, a column vector corresponding to the transposed key matrix corresponding to the key matrix is ​​input into the PE through the first cross network.

[0234] Optionally, the distribution network module 1102 is further configured to:

[0235] Based on the column index of the PE, the corresponding row vector in the value matrix is ​​input into the PE through the second cross network.

[0236] Optionally, the distribution network module 1102 is specifically configured to:

[0237] The row elements in the corresponding row vectors in the value matrix are input into the PE periodically through the second cross network according to the number of cycles, and the number of cycles is equal to the number of columns in the value matrix.

[0238] The sparse index preprocessing module 1101 is specifically used to:

[0239] Dividing the sparse matrix into blocks to obtain the sparse matrix blocks;

[0240] Obtaining the number of non-zero elements and the index coordinates of the non-zero elements in the sparse matrix block;

[0241] The number of non-zero elements and the index coordinates are arranged in one dimension to obtain the sparse index corresponding to the sparse matrix block.

[0242] Thus, embodiments of the present application provide a data processing device for sparse matrix multiplication. In its specific implementation, the device can be considered a computational structure consisting of a sparse index preprocessing module, a distribution network module, a one-dimensional systolic array module, and a group summation network module. For a sparse matrix input, the sparse index preprocessing module first arranges the sparse data into a tightly connected one-dimensional format. Then, some of the sparse data passes through the distribution network module and is fed into a one-dimensional systolic array module for computation. The one-dimensional systolic array consists of multiple multiplier-accumulators. When computing matrix inner products, the multiplier results are accumulated in the array; when computing matrix row products, the multiplier results are not accumulated in the array but are directly fed into the group summation network module for summation. Matrix inner products are suitable for the SDDMM operation of the attention mechanism in the Transformer model, while matrix row products are suitable for the SpMM operation therein. In other words, the solution provided by embodiments of the present application supports both processing methods simultaneously, saving hardware resources and reducing the storage and handling overhead of intermediate results. The computational structure also supports GEMM operations, offering high scalability and versatility.

[0243] It should be noted that the specific implementation of the device 1100 can be found in Figure 12 The overall process execution diagram of a data processing device 1100 for sparse matrix multiplication calculation is shown in FIG. The technical effect achieved by the device 1100 can be seen in FIG. Figure 1 The relevant description in the method shown.

[0244] Among them, for Figure 12 The matrix block part and the summation module are explained in the figure. Because for each sparse matrix block, the number of non-zero elements in it may not be less than or equal to the preset data of PE in the one-dimensional systolic array, it is necessary to sum the output matrix of each calculation to finally obtain the output matrix Z corresponding to the sparse matrix block.

[0245] For ease of understanding, the following two situations are explained:

[0246] In the embodiment of the present application, the number of 4 non-zero elements in the sparse matrix block A is equal to the number of PEs in the one-dimensional systolic matrix. The result after grouping and summing is: 0p =[2.70,3.28,1.33,2.67,2.69,1.45],z 1p =[0.25, 0.40, 0.15, 0.45, 0.20, 0.30] and z 2p =[0.65, 1.05, 0.45, 0.80, 0.55, 0.35] is the final output result.

[0247] However, for the 5 non-zero elements in the sparse matrix block B, there are only 4 PEs in the one-dimensional systolic array, so the calculation cannot be completed at one time. Then, first calculate the output matrix Z corresponding to (0, 1), (1, 2), (2, 1), and (2, 3) 1 , and then calculate the output matrix Z corresponding to (3, 2) 2 , through the matrix block part and summation module, Z 1 and Z 2 Sum and finally get the output matrix Z.

[0248] In addition, the embodiment of the present application further provides an electronic device 1300, such as Figure 13 As shown, the electronic device 1300 includes a processor 1301 and a memory 1302:

[0249] The memory 1302 is used to store computer programs;

[0250] The processor 1301 is configured to execute according to the computer program Figure 1 Provided method.

[0251] It should be noted that, in actual implementation, the processor 1301 may be a processor in a Transformer model or an accelerator in a Transformer model.

[0252] In addition, an embodiment of the present application further provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the method provided by the embodiment of the present application.

[0253] Through the description of the above embodiments, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above embodiment methods can be implemented by means of software plus a general hardware platform. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment or certain parts of the embodiments of the present application.

[0254] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment. The device embodiment described above is merely illustrative, in which the modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the goals of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0255] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. It should be noted that those skilled in the art can make several improvements and modifications without departing from the scope of protection of the present application, and such improvements and modifications should also be considered as within the scope of protection of the present application.

Claims

1. A data processing method for sparse matrix multiplication calculation, characterized in that: The method comprises: Preprocessing the sparse matrix to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; the sparse index includes the number of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements; assigning corresponding index coordinates to computing elements (PEs) in a one-dimensional systolic array according to the number of non-zero elements in the sparse index; the index coordinates comprising a row index and a column index; the one-dimensional systolic array comprising a predetermined number of PEs arranged in parallel; the PEs being reconfigurable PEs comprising a first computing path and a second computing path; the first computing path being configured to perform a matrix inner product operation, and the second computing path being configured to perform a matrix row product operation; performing a multiplication-accumulation calculation on a corresponding row vector in a query matrix and a corresponding column vector in a transposed key matrix corresponding to a key matrix according to the first calculation path in the PE, and outputting a calculation result corresponding to the PE; the corresponding row vector in the query matrix corresponds to the row index of the PE, and the corresponding column vector in the transposed key matrix corresponding to the key matrix corresponds to the column index of the PE; performing a multiplication operation on the calculation result and a corresponding row vector in a value matrix according to the second calculation path in the PE, and outputting a multiplication result corresponding to the PE; the corresponding row vector in the value matrix corresponds to the column index of the PE; Performing group summation on the multiplication results having the number of non-zero elements through a group summation network to obtain at least one group summation result; At least one of the grouped summation results is reordered according to a row index corresponding to the grouped summation result to obtain an output matrix; the output matrix corresponds to the sparse matrix block.

2. The method according to claim 1, characterized in that The first calculation path includes a first multiplexer, a second multiplexer, a third multiplexer, a fourth multiplexer, a multiplier, and an adder. The performing of multiplication and accumulation calculations on the corresponding row vectors in the query matrix and the corresponding column vectors in the transposed key matrix corresponding to the key matrix according to the first calculation path in the PE, and outputting the calculation results corresponding to the PE, includes: Traversing the row elements in the corresponding row vector of the query matrix and the column elements in the corresponding column vector of the transposed key matrix corresponding to the key matrix; the column index of the row element corresponds to the row index of the column element; Inputting the row elements into the multiplier through the first multiplexer; Input the column elements into the multiplier through the second multiplexer; Performing a multiplication calculation on the row element and the row element according to the multiplier to obtain an element calculation result; calling the third multiplexer to input the existing element calculation result and the sum into the adder; the existing element calculation result and the sum of the element calculation results that have been accumulated by the adder; Accumulate the element calculation result and the existing element calculation results according to the adder to obtain the calculation result; The calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

3. The method according to claim 1, characterized in that The second calculation path includes a first multiplexer, a second multiplexer, a fourth multiplexer, and a multiplier. The performing of a multiplication calculation on the calculation result and a corresponding row vector in the value matrix according to the second calculation path in the PE, and outputting a multiplication result corresponding to the PE, includes: inputting the calculation result into the multiplier through the first multiplexer; Inputting the corresponding row vector in the value matrix into the multiplier through the second multiplexer; performing a multiplication calculation on the calculation result and a corresponding row vector in the value matrix according to the multiplier to obtain the multiplication result; The multiplication result is output through the fourth multiplexer to complete the multiplication SpMM of the sparse matrix and the dense matrix.

4. The method according to claim 1, wherein Before performing multiplication and accumulation calculations on the corresponding row vectors in the query matrix and the corresponding column vectors in the transposed key matrix corresponding to the key matrix according to the first calculation path in the PE, and outputting the calculation results corresponding to the PE, the method further includes: Based on the row index of the PE, the corresponding row vector in the query matrix is ​​input into the PE through the first cross network; Based on the column index of the PE, a column vector corresponding to the transposed key matrix corresponding to the key matrix is ​​input into the PE through the first cross network.

5. The method according to claim 1, wherein Before performing a multiplication calculation on the calculation result and the corresponding row vector in the value matrix according to the second calculation path in the PE and outputting the multiplication result corresponding to the PE, the method further includes: Based on the column index of the PE, the corresponding row vector in the value matrix is ​​input into the PE through the second cross network.

6. The method according to claim 5, characterized in that The step of inputting the corresponding row vector in the value matrix into the PE through the second cross network includes: The row elements in the corresponding row vectors in the value matrix are input into the PE periodically through the second cross network according to the number of cycles, and the number of cycles is equal to the number of columns in the value matrix.

7. The method according to claim 1, characterized in that The preprocessing of the sparse matrix to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block includes: Dividing the sparse matrix into blocks to obtain the sparse matrix blocks; Obtaining the number of non-zero elements and the index coordinates of the non-zero elements in the sparse matrix block; The number of non-zero elements and the index coordinates are arranged in one dimension to obtain the sparse index corresponding to the sparse matrix block.

8. A data processing device for sparse matrix multiplication calculation, characterized in that: The device comprises: a sparse index pre-processing module, a distribution network module, a one-dimensional systolic array module and a group summation network module; The sparse index preprocessing module is used to preprocess the sparse matrix to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; the sparse index includes the number of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements; The distribution network module is configured to allocate corresponding index coordinates to computing elements (PEs) in a one-dimensional systolic array according to the number of non-zero elements in the sparse index; the index coordinates include a row index and a column index; the one-dimensional systolic array includes a preset number of PEs arranged in parallel; the PEs are reconfigurable PEs including a first computing path and a second computing path; the first computing path is configured to perform a matrix inner product operation, and the second computing path is configured to perform a matrix row product operation; The one-dimensional systolic array module is configured to perform a multiplication-accumulation calculation on a corresponding row vector in a query matrix and a corresponding column vector in a transposed key matrix corresponding to a key matrix according to the first calculation path in the PE, and output a calculation result corresponding to the PE; the corresponding row vector in the query matrix corresponds to the row index of the PE, and the corresponding column vector in the transposed key matrix corresponding to the key matrix corresponds to the column index of the PE; The one-dimensional systolic array module is further configured to perform a multiplication operation on the calculation result and a corresponding row vector in a value matrix according to the second calculation path in the PE, and output a multiplication result corresponding to the PE; the corresponding row vector in the value matrix corresponds to the column index of the PE; The group summation network module is used to perform group summation on the multiplication results of the number of non-zero elements through a group summation network to obtain at least one group summation result; The group summation network module is further used to reorder at least one of the group summation results according to the row index corresponding to the group summation result to obtain an output matrix; the output matrix corresponds to the sparse matrix block.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory: The memory is used to store computer programs; The processor is configured to execute the method according to any one of claims 1 to 7 according to the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method according to any one of claims 1 to 7.

Images