Data processing method and device for sparse matrix multiplication calculation, equipment and medium

Abstract

The invention discloses a data processing method and device for sparse matrix multiplication, equipment and a medium, and the method comprises the steps: carrying out the preprocessing of a sparse matrix, and obtaining a sparse matrix block and a sparse index corresponding to the sparse matrix block; according to the number of non-zero elements in the sparse index, corresponding index coordinates are distributed to a calculation unit PE in the one-dimensional systolic array; performing multiply-accumulate calculation according to a first calculation path in the PE, and outputting a calculation result; performing multiplication calculation according to a second calculation path in the PE, and outputting a multiplication result corresponding to the PE; performing packet summation through a packet summation network to obtain at least one packet summation result; and reordering the at least one group summation result to obtain an output matrix. Therefore, two types of sparse matrix multiplication of SDDMM and SpMM are efficiently processed by fully utilizing the characteristics of the matrix inner product and the matrix row product, and the intermediate calculation result is temporarily stored and reused, so that the data movement and storage overhead is reduced, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment and medium for sparse matrix multiplication calculation. Background Art

[0002] Transformer is a deep learning model based on the attention mechanism, which has achieved remarkable results in fields such as natural language processing. In the field of deep learning, especially in calculations involving the Transformer model, matrix multiplication is one of the core operations.

[0003] However, as the scale of the model increases, traditional dense matrix multiplication faces problems such as high consumption of computing resources and long computing time. As an optimization method, sparse matrix multiplication reduces unnecessary calculations by utilizing the large number of zero elements or negligible elements in the matrix, thereby improving data processing efficiency.

[0004] However, after a dense matrix is ​​made sparse, the non-zero elements therein are usually irregularly distributed, which makes it impossible for traditional processors to efficiently process the data of the sparse matrix. That is, the data processing efficiency of sparse matrix multiplication in the prior art is low. Summary of the invention

[0005] The present application provides a data processing method, device, equipment and medium for sparse matrix multiplication calculation, which can improve the processing efficiency of irregular sparse matrices.

[0006] In a first aspect, the present application provides a data processing method for sparse matrix multiplication calculation, the method comprising:

[0007] Preprocessing the sparse matrix to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; the sparse index includes the number of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements;

[0008] According to the number of non-zero elements in the sparse index, the corresponding index coordinates are allocated to computing units PE in a one-dimensional systolic array; the index coordinates include a row index and a column index; the one-dimensional systolic array includes a preset number of PEs arranged in parallel; the PE is a reconfigurable PE including a first computing path and a second computing path; the first computing path is used to perform a matrix inner product operation, and the second computing path is used to perform a matrix row product operation;

[0009] According to the first calculation path in the PE, perform multiplication and accumulation calculations on the corresponding row vectors in the query matrix and the corresponding column vectors in the transposed key matrix corresponding to the key matrix, and output the calculation results corresponding to the PE; the corresponding row vectors in the query matrix correspond to the row index of the PE, and the corresponding column vectors in the transposed key matrix corresponding to the key matrix correspond to the column index of the PE;

[0010] According to the second calculation path in the PE, perform a multiplication calculation on the calculation result and a corresponding row vector in a value matrix, and output a multiplication result corresponding to the PE; the corresponding row vector in the value matrix corresponds to the column index of the PE;

[0011] Performing group summation on the multiplication results of the number of non-zero elements through a group summation network to obtain at least one group summation result;

[0012] At least one of the grouped summation results is reordered according to the row index corresponding to the grouped summation results to obtain an output matrix; the output matrix corresponds to the sparse matrix block.

[0013] Optionally, the first calculation path includes a first multiplexer, a second multiplexer, a third multiplexer, a fourth multiplexer, a multiplier and an adder, and the performing of multiplication and accumulation calculations on the corresponding row vectors in the query matrix and the corresponding column vectors in the transposed key matrix corresponding to the key matrix according to the first calculation path in the PE, and outputting the calculation results corresponding to the PE includes:

[0014] Traversing the row elements in the corresponding row vector in the query matrix and the column elements in the corresponding column vector in the transposed key matrix corresponding to the key matrix; the column index of the row element corresponds to the row index of the column element;

[0015] Input the row elements into the multiplier through the first multiplexer;

[0016] Input the column elements into the multiplier through the second multiplexer;

[0017] Performing multiplication calculation on the row element and the row element according to the multiplier to obtain an element calculation result;

[0018] Calling the third multiplexer to input the existing element calculation result and the sum into the adder; the existing element calculation result and the sum of the element calculation results that have been accumulated and calculated by the adder;

[0019] Accumulate the element calculation result and the existing element calculation result according to the adder to obtain the calculation result;

[0020] The calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0021] Optionally, the second calculation path includes a first multiplexer, a second multiplexer, a fourth multiplexer and a multiplier, and the performing multiplication calculation on the calculation result and a corresponding row vector in the value matrix according to the second calculation path in the PE, and outputting the multiplication result corresponding to the PE includes:

[0022] Inputting the calculation result into the multiplier through the first multiplexer;

[0023] Inputting the corresponding row vector in the value matrix into the multiplier through the second multiplexer;

[0024] Performing a multiplication calculation on the calculation result and a corresponding row vector in the value matrix according to the multiplier to obtain the multiplication result;

[0025] The multiplication result is output through the fourth multiplexer to complete the multiplication SpMM of the sparse matrix and the dense matrix.

[0026] Optionally, before performing multiplication and accumulation calculations on the corresponding row vectors in the query matrix and the corresponding column vectors in the transposed key matrix corresponding to the key matrix according to the first calculation path in the PE, and outputting the calculation results corresponding to the PE, the method further includes:

[0027] Based on the row index of the PE, input the corresponding row vector in the query matrix into the PE through the first cross network;

[0028] Based on the column index of the PE, a column vector corresponding to the transposed key matrix corresponding to the key matrix is ​​input into the PE through the first cross network.

[0029] Optionally, before performing a multiplication calculation on the calculation result and a corresponding row vector in the value matrix according to the second calculation path in the PE and outputting the multiplication result corresponding to the PE, the method further includes:

[0030] Based on the column index of the PE, the corresponding row vector in the value matrix is ​​input into the PE through a second cross network.

[0031] Optionally, inputting the corresponding row vector in the value matrix into the PE through a second cross network includes:

[0032] The row elements in the corresponding row vector in the value matrix are input into the PE through the second cross network cycle by cycle according to the number of cycles, and the number of cycles is equal to the number of columns of the value matrix.

[0033] Optionally, preprocessing the sparse matrix to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block includes:

[0034] Divide the sparse matrix into blocks to obtain the sparse matrix blocks;

[0035] Obtaining the number of non-zero elements and the index coordinates of the non-zero elements in the sparse matrix block;

[0036] The number of non-zero elements and the index coordinates are arranged in one dimension to obtain the sparse index corresponding to the sparse matrix block.

[0037] Optionally, outputting the calculation result through the fourth multiplexer to complete a sampled dense-dense matrix multiplication SDDMM includes:

[0038] Determine whether the row element is the last row element in the corresponding row vector in the query matrix;

[0039] If the row element is not the last row element in the row vector corresponding to the query matrix, then return to execute the traversal of the row elements in the row vector corresponding to the query matrix and the column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix until the last row element in the row vector corresponding to the query matrix is ​​obtained;

[0040] If the row element is the last row element in the corresponding row vector in the query matrix, the calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0041] Optionally, outputting the calculation result through the fourth multiplexer to complete a sampled dense-dense matrix multiplication SDDMM includes:

[0042] Determine whether the column element is the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix;

[0043] If the column element is not the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix, then returning to execute the traversal of the row elements in the corresponding row vector in the query matrix and the column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix until the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix is ​​obtained;

[0044] If the column element is the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix, the calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0045] In a second aspect, the present application further provides a data processing device for sparse matrix multiplication calculation, the device comprising: a sparse index preprocessing module, a distribution network module, a one-dimensional systolic array module and a group summation network module;

[0046] The sparse index preprocessing module is used to preprocess the sparse matrix to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; the sparse index includes the number of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements;

[0047] The distribution network module is used to allocate the corresponding index coordinates to the computing units PE in the one-dimensional systolic array according to the number of non-zero elements in the sparse index; the index coordinates include a row index and a column index; the one-dimensional systolic array includes a preset number of PEs arranged in parallel; the PE is a reconfigurable PE including a first computing path and a second computing path; the first computing path is used to perform a matrix inner product operation, and the second computing path is used to perform a matrix row product operation;

[0048] The one-dimensional systolic array module is used to perform multiplication and accumulation calculations on the corresponding row vectors in the query matrix and the corresponding column vectors in the transposed key matrix corresponding to the key matrix according to the first calculation path in the PE, and output the calculation results corresponding to the PE; the corresponding row vectors in the query matrix correspond to the row index of the PE, and the corresponding column vectors in the transposed key matrix corresponding to the key matrix correspond to the column index of the PE;

[0049] The one-dimensional systolic array module is further used to perform a multiplication calculation on the calculation result and a corresponding row vector in a value matrix according to the second calculation path in the PE, and output a multiplication result corresponding to the PE; the corresponding row vector in the value matrix corresponds to the column index of the PE;

[0050] The group summation network module is used to perform group summation on the multiplication results of the number of non-zero elements through a group summation network to obtain at least one group summation result;

[0051] The group summation network module is further used to reorder at least one of the group summation results according to the row index corresponding to the group summation result to obtain an output matrix; the output matrix corresponds to the sparse matrix block.

[0052] Optionally, the first computing path includes a first multiplexer, a second multiplexer, a third multiplexer, a fourth multiplexer, a multiplier and an adder, and the one-dimensional systolic array module is specifically used for:

[0053] Traversing the row elements in the corresponding row vector in the query matrix and the column elements in the corresponding column vector in the transposed key matrix corresponding to the key matrix; the column index of the row element corresponds to the row index of the column element;

[0054] Input the row elements into the multiplier through the first multiplexer;

[0055] Input the column elements into the multiplier through the second multiplexer;

[0056] Performing multiplication calculation on the row element and the row element according to the multiplier to obtain an element calculation result;

[0057] Calling the third multiplexer to input the existing element calculation result and the sum into the adder; the existing element calculation result and the sum of the element calculation results that have been accumulated and calculated by the adder;

[0058] Accumulate the element calculation result and the existing element calculation result according to the adder to obtain the calculation result;

[0059] The calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0060] Optionally, the second computing path includes a first multiplexer, a second multiplexer, a fourth multiplexer and a multiplier, and the one-dimensional systolic array module is specifically used for:

[0061] Inputting the calculation result into the multiplier through the first multiplexer;

[0062] Inputting the corresponding row vector in the value matrix into the multiplier through the second multiplexer;

[0063] Performing a multiplication calculation on the calculation result and a corresponding row vector in the value matrix according to the multiplier to obtain the multiplication result;

[0064] The multiplication result is output through the fourth multiplexer to complete the multiplication SpMM of the sparse matrix and the dense matrix.

[0065] Optionally, the one-dimensional systolic array module is specifically used for:

[0066] Determine whether the row element is the last row element in the corresponding row vector in the query matrix;

[0067] If the row element is not the last row element in the row vector corresponding to the query matrix, then return to execute the traversal of the row elements in the row vector corresponding to the query matrix and the column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix until the last row element in the row vector corresponding to the query matrix is ​​obtained;

[0068] If the row element is the last row element in the corresponding row vector in the query matrix, the calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0069] Optionally, the one-dimensional systolic array module is specifically used for:

[0070] Determine whether the column element is the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix;

[0071] If the column element is not the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix, then returning to execute the traversal of the row elements in the corresponding row vector in the query matrix and the column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix until the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix is ​​obtained;

[0072] If the column element is the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix, the calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0073] Optionally, the distribution network module is further used to:

[0074] Based on the row index of the PE, input the corresponding row vector in the query matrix into the PE through the first cross network;

[0075] Based on the column index of the PE, a column vector corresponding to the transposed key matrix corresponding to the key matrix is ​​input into the PE through the first cross network.

[0076] Optionally, the distribution network module is further used to:

[0077] Based on the column index of the PE, the corresponding row vector in the value matrix is ​​input into the PE through a second cross network.

[0078] Optionally, the distribution network module is specifically used to:

[0079] The row elements in the corresponding row vector in the value matrix are input into the PE through the second cross network cycle by cycle according to the number of cycles, and the number of cycles is equal to the number of columns of the value matrix.

[0080] The sparse index preprocessing module is specifically used for:

[0081] Divide the sparse matrix into blocks to obtain the sparse matrix blocks;

[0082] Obtaining the number of non-zero elements and the index coordinates of the non-zero elements in the sparse matrix block;

[0083] The number of non-zero elements and the index coordinates are arranged in one dimension to obtain the sparse index corresponding to the sparse matrix block.

[0084] In a third aspect, the present application further provides an electronic device, the electronic device comprising a processor and a memory:

[0085] The memory is used to store computer programs;

[0086] The processor is used to execute the method provided in the first aspect according to the computer program.

[0087] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method provided in the first aspect above.

[0088] It can be seen that this application has the following beneficial effects:

[0089] The present application provides a data processing method, device, equipment and medium for sparse matrix multiplication calculation. In the method, a sparse matrix is ​​preprocessed to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; the sparse index includes the number of non-zero elements of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements; according to the number of non-zero elements in the sparse index, a corresponding index coordinate is allocated to a computing unit PE in a one-dimensional systolic array; the index coordinate includes a row index and a column index; the one-dimensional systolic array includes a preset number of PEs arranged in parallel; the PE is a reconfigurable PE including a first computing path and a second computing path; the first computing path is used to perform a matrix inner product operation, and the second computing path is used to perform a matrix row product operation; according to the first computing path in the PE ... The corresponding row vector and the corresponding column vector in the transposed key matrix corresponding to the key matrix are multiplied and accumulated, and the calculation result corresponding to PE is output; the corresponding row vector in the query matrix corresponds to the row index of PE, and the corresponding column vector in the transposed key matrix corresponding to the key matrix corresponds to the column index of PE; according to the second calculation path in PE, the calculation result and the row vector corresponding to the column index in the value matrix are multiplied, and the multiplication result corresponding to PE is output; the corresponding row vector in the value matrix corresponds to the column index of PE; the multiplication results of the number of non-zero elements are grouped and summed through the group summation network to obtain at least one group summation result; according to the row index corresponding to the group summation result, at least one group summation result is reordered to obtain the output matrix; the output matrix corresponds to the sparse matrix block.

[0090] Thus, the method provided by the embodiment of the present application, on the one hand, targets the irregular sparsity of the sparse matrix, and makes the irregular elements closely connected by preprocessing the sparse matrix, effectively dealing with the problem of irregular distribution of elements in the sparse matrix, and improving the processing capability of the irregular sparse matrix. At the same time, directly storing the non-zero elements in the sparse matrix block in the form of index coordinates not only simplifies the hardware logic of decoding the sparse format and improves the data processing efficiency, but also can quickly obtain the index coordinates of the required elements, improve the efficiency of data access, and thus improve the data processing efficiency. On the other hand, compared with the two-dimensionally arranged PEs in the prior art, the use of PEs arranged in parallel in a one-dimensional systolic array can reduce the idle rate of PEs during the calculation process, make full use of the parallelism and scalability of the hardware, improve the utilization rate of hardware resources, and improve data processing efficiency under limited hardware resources. On the other hand, the matrix inner product operation is realized through the first calculation path in the PE, and the matrix row product operation is realized through the second calculation path in the PE. The characteristics of the matrix inner product and the matrix row product are fully utilized to efficiently process the two types of sparse matrix multiplications, SDDMM and SpMM. In addition, because the calculation results between SDDMM and SpMM are temporarily stored and reused, the overhead of storing and repeatedly moving the intermediate calculation results is saved, thereby improving the data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0092] Figure 1 A schematic diagram of a flow chart of a data processing method for sparse matrix multiplication calculation in an embodiment of the present application;

[0093] Figure 2 A schematic diagram of sparse matrix preprocessing provided in an embodiment of the present application;

[0094] Figure 3 A schematic diagram of the overall process of a data processing method for sparse matrix multiplication calculation provided in an embodiment of the present application;

[0095] Figure 4 A schematic diagram of inputting a one-dimensional systolic array through a distribution network provided in an embodiment of the present application;

[0096] Figure 5 A schematic diagram of the overall structure of a reconfigurable PE provided in an embodiment of the present application;

[0097] Figure 6 A schematic diagram of the structure of the first computing path of the reconfigurable PE provided in an embodiment of the present application;

[0098] Figure 7 A schematic diagram of an example of a calculation process of a matrix inner product and a matrix outer product provided in an embodiment of the present application;

[0099] Figure 8 A schematic diagram of the structure of the second computing path of the reconfigurable PE provided in an embodiment of the present application;

[0100] Fig. 9 A schematic diagram of a group summation process provided in an embodiment of the present application;

[0101] Fig.10 A comparison chart of computing efficiency under different sparsity provided in the embodiment of the present application;

[0102] Fig.11 A schematic diagram of the structure of a data processing device 1100 for sparse matrix multiplication calculation provided in an embodiment of the present application;

[0103] Fig.12 A schematic diagram of the overall process execution of a data processing device 1100 for sparse matrix multiplication calculation provided in an embodiment of the present application;

[0104] Fig.13A schematic diagram of the structure of an electronic device 1300 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0105] The multiple involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first", "second", etc. are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0106] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It is understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. In addition, it should be noted that, for ease of description, only the parts related to the present application are shown in the accompanying drawings, not all structures.

[0107] At present, with the increase of model scale, traditional dense matrix multiplication faces problems such as high consumption of computing resources and long computing time. As an optimization method, sparse matrix multiplication reduces unnecessary calculations by utilizing the large number of zero elements or negligible elements in the matrix, thereby improving data processing efficiency.

[0108] However, after a dense matrix is ​​made sparse, its non-zero elements are often irregularly distributed, which makes it impossible for traditional processors to efficiently process sparse matrix data. That is, the data processing efficiency of sparse matrix multiplication in the prior art is low.

[0109] The applicant has found that the reasons why traditional processors cannot efficiently process sparse matrix data are as follows:

[0110] ① For example, the computational complexity of the Transformer model is extremely high, especially the matrix multiplication operation in the self-attention mechanism uses a two-dimensional PE computing unit. The calculation process of the two-dimensional PE computing unit occupies a large amount of computing resources and time, and cannot give full play to the performance of the hardware, resulting in low data processing efficiency.

[0111] ② With the trend of Transformer model becoming sparse, sparse matrix multiplication becomes the key to improving computational efficiency. However, there are many problems with existing sparse matrix processing methods. On the one hand, irregular sparse methods affect the efficiency of hardware execution; on the other hand, regular sparse methods often require retraining the model, which brings additional training overhead. Among them, in irregular sparse methods, the number of non-zero elements in each row and column of the sparse matrix is ​​not fixed. The matrix elements of this method are unevenly distributed, and some rows or columns may contain more non-zero elements; among them, in regular sparse methods, the number of non-zero elements in each row and column of the sparse matrix is ​​fixed. This means that the elements in the matrix are evenly distributed, and the frequency of occurrence of each element is roughly the same.

[0112] ③ In the deep learning computing process, data sources and forms are diverse, and data generated at different stages have different characteristics and requirements. For example, in the sparse attention mechanism, multiple data streams are involved, including full-precision data, quantized data, pruned and sorted data, etc. Existing technologies cannot integrate these complex data streams well, resulting in inefficient data transmission, waste of computing resources, and low data processing efficiency.

[0113] In summary, in order to solve the problems of low data processing efficiency caused by low hardware resource utilization, low adaptability of sparse matrices, poor integration of complex data flows, etc. corresponding to the Transformer model deep learning model, the embodiments of the present application provide a data processing method, device, equipment and medium for sparse matrix multiplication calculation. In the method, the sparse matrix is ​​preprocessed to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; the sparse index includes the number of non-zero elements of the non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements; according to the number of non-zero elements in the sparse index, the corresponding index coordinates are assigned to the computing unit PE in the one-dimensional systolic array; the index coordinates include row index and column index; the one-dimensional systolic array includes a preset number of PEs arranged in parallel; the PE is a reconfigurable PE including a first computing path and a second computing path; the first computing path is used to perform matrix inner product Operation, the second calculation path is used to perform matrix row product operation; according to the first calculation path in PE, the corresponding row vector in the query matrix and the corresponding column vector in the transposed key matrix corresponding to the key matrix are multiplied and accumulated, and the calculation result corresponding to PE is output; the corresponding row vector in the query matrix corresponds to the row index of PE, and the corresponding column vector in the transposed key matrix corresponding to the key matrix corresponds to the column index of PE; according to the second calculation path in PE, the calculation result and the row vector corresponding to the column index in the value matrix are multiplied, and the multiplication result corresponding to PE is output; the corresponding row vector in the value matrix corresponds to the column index of PE; the multiplication results of the number of non-zero elements are grouped and summed through the group summation network to obtain at least one group summation result; according to the row index corresponding to the group summation result, at least one group summation result is reordered to obtain an output matrix; the output matrix corresponds to the sparse matrix block.

[0114] Thus, the method provided by the embodiment of the present application, on the one hand, for the irregular sparsity of the sparse matrix, the irregular elements can be closely connected by preprocessing the sparse matrix, effectively coping with the problem of irregular distribution of elements in the sparse matrix, and improving the processing capability of the irregular sparse matrix. At the same time, directly storing the non-zero elements in the sparse matrix block in the form of index coordinates not only simplifies the hardware logic of decoding the sparse format and improves the data processing efficiency, but also can quickly obtain the index coordinates of the required elements, improves the efficiency of data access, and thus improves the data processing efficiency. On the other hand, the matrix inner product operation is realized through the first calculation path in the PE, and the matrix row product operation is realized through the second calculation path in the PE, making full use of the characteristics of the matrix inner product and the matrix row product to efficiently process the two types of sparse matrix multiplications of SDDMM and SpMM, realizing the sparse matrix multiplication calculation that integrates multiple data streams, and because the calculation results between SDDMM and SpMM are temporarily stored and reused, the overhead of storing and repeatedly transporting the intermediate calculation results is saved, and the data processing efficiency is improved. On the other hand, compared with the two-dimensionally arranged PE in the prior art, the embodiment of the present application uses computing units PE arranged in parallel in a one-dimensional systolic array for calculation, which makes full use of the parallelism and scalability of the hardware, can improve the utilization rate of hardware resources, and improve data processing efficiency under limited hardware resources.

[0115] To facilitate understanding of the specific implementation of the data processing method for sparse matrix multiplication calculation provided in the embodiment of the present application, it will be explained below with reference to the accompanying drawings.

[0116] It should be noted that the subject implementing the data processing method for sparse matrix multiplication calculation may be a data processing device for sparse matrix multiplication calculation provided in the embodiment of the present application, and the data processing device for sparse matrix multiplication calculation may be carried in an electronic device or a functional module of an electronic device. The electronic device in the embodiment of the present application may be any device capable of implementing the data processing method for sparse matrix multiplication calculation in the embodiment of the present application, for example, an Internet of Things (IoT) device.

[0117] See also Figure 1 , a flow chart of a data processing method for sparse matrix multiplication calculation is provided, in an embodiment of the present application, for example, the following steps may be included:

[0118] S1, preprocessing the sparse matrix to obtain sparse matrix blocks and sparse indexes corresponding to the sparse matrix blocks.

[0119] It should be noted that the sparse index includes the number of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements. A sparse matrix refers to a data set in which most elements in the two-dimensional table are zero and there are relatively few non-zero elements.

[0120] In a possible implementation, S1 provided in the embodiment of the present application may include:

[0121] S101, dividing the sparse matrix into blocks to obtain sparse matrix blocks.

[0122] S102, obtaining the number of non-zero elements and the index coordinates of the non-zero elements in the sparse matrix block.

[0123] S103, performing one-dimensional arrangement on the number of non-zero elements and the index coordinates to obtain a sparse index corresponding to the sparse matrix block.

[0124] It should be noted that the specific execution process of steps S101 to S103 can be found in Figure 2 , Figure 2 A schematic diagram of sparse matrix preprocessing is provided.

[0125] exist Figure 2 In the sparse matrix preprocessing process shown, first, the sparse matrix is ​​divided into blocks, the sparse matrix is ​​an 8×8 matrix, and the sparse matrix is ​​divided into 6 sparse matrix blocks; then, for each sparse matrix block, the number of non-zero elements and the index coordinates of the non-zero elements in the sparse matrix block are obtained, and the number of non-zero elements and the index coordinates are arranged in one dimension to obtain the sparse index of the sparse matrix block, wherein the white squares in the sparse matrix represent zero elements, and the non-white squares in the sparse matrix represent non-zero elements, and the generated index is stored in the on-chip cache, thereby improving the speed of acquiring data. In the embodiment of the present application, the number of non-zero elements in the sparse matrix block A is obtained to be 4, and the index coordinates of the non-zero elements are (0, 0), (0, 2), (1, 1) and (2, 3), respectively, that is, the sparse index includes the index coordinates (0, 0), (0, 2), (1, 1), (2, 3) and the number of non-zero elements 4.

[0126] In the specific implementation process, the sparse matrix can be divided into blocks according to the number of PEs in the one-dimensional systolic array (see the one-dimensional systolic array in step S2 for details. The one-dimensional systolic array provided in the embodiment of the present application includes 4 PEs). The number of PEs in the one-dimensional systolic array is equal to the number of columns of the sparse matrix block. The coordinates of the non-zero elements are read in one by one and judged, and they are placed in the corresponding blocks through a first-in-first-out queue. Among them, the embodiment of the present application does not limit the number of rows of the sparse matrix block when dividing the blocks, that is, the 6 sparse matrix blocks divided by the sparse matrix are: 3×4 matrix, 3×4 matrix, 3×4 matrix, 3×4 matrix, 2×4 matrix and 2×4 matrix from top to bottom and from left to right.

[0127] In an embodiment of the present application, by dividing the sparse matrix into blocks and arranging it in one dimension, irregular data can be tightly connected to facilitate subsequent calculations. This preprocessing method can effectively deal with the problem of irregular distribution of elements in the sparse matrix and improve the processing capability of irregular sparse matrices.

[0128] Furthermore, in the embodiment of the present application, the COO format is used to directly store the non-zero elements in the sparse matrix block in the form of index coordinates, which reduces the storage quantity, quickly obtains the index coordinates of the required elements during the calculation process, improves the efficiency of data access, and thus improves the data processing efficiency. In addition, the direct storage of coordinates does not require decoding, simplifies the hardware logic of decoding the sparse format, and also improves the data processing efficiency to a certain extent.

[0129] S2, assign corresponding index coordinates to computing units PE in the one-dimensional systolic array according to the number of non-zero elements in the sparse index.

[0130] It should be noted that the index coordinates include row indices and column indices; the one-dimensional systolic array includes a preset number of PEs arranged in parallel, and the PE is a reconfigurable PE including a first computing path and a second computing path, the first computing path is used to perform matrix inner product operations, and the second computing path is used to perform matrix row product operations.

[0131] In the embodiment of the present application, the one-dimensional systolic array includes a preset number of PEs arranged in parallel, and the reconfigurable PE means that each PE is configured to calculate two modes: matrix inner product and matrix row product. The preset number can be set according to demand, and the embodiment of the application does not limit the preset number. In order to facilitate understanding of the solution of the embodiment of the present application, the preset number is 4, that is, the one-dimensional systolic array includes 4 PEs arranged in parallel.

[0132] In one possible implementation, see Figure 3 , Figure 3 A schematic diagram of the overall flow of a data processing method for sparse matrix multiplication calculation provided in an embodiment of the present application.

[0133] In the embodiment of the present application, the corresponding index coordinates are allocated to the PE in the one-dimensional systolic array according to the number of non-zero elements in the sparse index. In the above example, it can be seen that the sparse index corresponding to the sparse matrix block A includes index coordinates (0, 0), (0, 2), (1, 1), (2, 3) and the number of non-zero elements is 4. Then, the index coordinates (0, 0), (0, 2), (1, 1), (2, 3) of the non-zero elements to be processed are allocated to the 4 parallel PEs respectively, that is, PE0 corresponds to the index coordinate (0, 0), PE1 corresponds to the index coordinate (0, 2), PE2 corresponds to the index coordinate (1, 1), and PE3 corresponds to the index coordinate (2, 3), where c represents the column number variable.

[0134] It should be noted that in the above example, the number of non-zero elements in the sparse matrix block is exactly 4, which is exactly equal to the number of PEs arranged in parallel in the one-dimensional systolic array. If the number of non-zero elements in a sparse matrix block B is 5, and the coordinates are (0, 1), (1, 2), (2, 1), (2, 3) and (3, 2) respectively, then in the process of assigning index coordinates to PEs in the one-dimensional systolic array, the four PEs should first be assigned index coordinates of (0, 1), (1, 2), (2, 1), and (2, 3). After the PE completes the corresponding calculation, the index coordinate (3, 2) is assigned to one of the PEs. Usually, the assignment is made in sequence, that is, the index coordinate (3, 2) is assigned to PE0.

[0135] Compared with the prior art in which PEs are arranged in two dimensions, the one-dimensional systolic array provided in the embodiment of the present application arranges PEs in parallel, and allocates sparse coordinates to the PEs in the one-dimensional systolic array according to the number of non-zero elements in the sparse matrix block, thereby reducing the number of idle PEs in the calculation process, improving calculation efficiency, and reducing calculation costs.

[0136] S3, according to the first calculation path in PE, perform multiplication and accumulation calculations on the corresponding row vector in the query matrix and the corresponding column vector in the transposed key matrix corresponding to the key matrix, and output the calculation result corresponding to PE.

[0137] Among them, the corresponding row vector in the query matrix corresponds to the row index of PE, and the corresponding column vector in the transposed key matrix corresponding to the key matrix corresponds to the column index of PE.

[0138] It should be noted that the query matrix is ​​usually represented by Q, and the key matrix is ​​usually represented by K. Through the first calculation path of the PE, the corresponding row vectors in the query matrix Q input to the PE and the transposed matrix K corresponding to the key matrix K are calculated. T The corresponding column vector in performs multiplication and accumulation calculations and outputs the calculation results corresponding to the PE.

[0139] In a possible implementation, before S3, the embodiment of the present application may further include:

[0140] Based on the row index of PE, the corresponding row vector in the query matrix is ​​input into PE through the first cross network;

[0141] Based on the column index of PE, the corresponding column vector in the transposed key matrix corresponding to the key matrix is ​​input into PE through the first cross network.

[0142] For specific implementation process, please refer to Figure 4 A schematic diagram of a one-dimensional systolic array input through a distribution network is shown. In the embodiment of the present application, based on the index coordinates corresponding to PE, the corresponding row vector in the query matrix Q and the transposed matrix K corresponding to the key matrix K are converted intoT The corresponding column vector in is input into the corresponding PE through the first cross network. Wherein, the query matrix Q and the key matrix K are both dense matrices, the query matrix Q is the left matrix, the key matrix K is the right matrix, and the PE is a reconfigurable PE.

[0143] In a possible implementation, the first computing path includes a first multiplexer, a second multiplexer, a third multiplexer, a fourth multiplexer, a multiplier, and an adder. For the specific structure of PE, please refer to Figure 5 , Figure 5 The overall structural diagram of the reconfigurable PE provided in the embodiment of the present application, wherein the first multiplexer is represented by M1, the second multiplexer is represented by M2, the third multiplexer is represented by M3, and the fourth multiplexer is represented by M4, that is, the first computing path includes: the first multiplexer M1, the second multiplexer M2, the third multiplexer M3, the fourth multiplexer M4, a multiplier and an adder, and 16 or 32 represents the data bit width, that is, 16 bits or 32 bits.

[0144] The specific execution process of S3 provided in the embodiment of this application can be found in Figure 6 and Figure 7 , Figure 6 A schematic diagram of the structure of the first computing path of the reconfigurable PE provided in an embodiment of the present application, Figure 7 A schematic diagram of an example of a calculation process of a matrix inner product and a matrix outer product provided in an embodiment of the present application, S3 may include:

[0145] S31, traverse the row elements in the corresponding row vector in the query matrix and the column elements in the corresponding column vector in the transposed key matrix corresponding to the key matrix.

[0146] Among them, the column index of the row element corresponds to the row index of the column element.

[0147] S32, input the row elements into the multiplier through the first multiplexer.

[0148] S33, input the column elements into the multiplier through the second multiplexer.

[0149] S34, performing multiplication calculation on row elements and row elements according to the multiplier to obtain element calculation results.

[0150] S35, calling the third multiplexer to calculate the existing elements and input the result to the adder.

[0151] The sum of the existing element calculation results is the sum of the element calculation results that have been accumulated by the adder.

[0152] S36, performing cumulative calculation on the element calculation result and the existing element calculation result according to the adder to obtain a calculation result;

[0153] S37, outputting the calculation result through the fourth multiplexer to complete the sampled dense-dense matrix multiplication (SDDMM).

[0154] Among them, SDDMM refers to the sparse matrix obtained by multiplying some rows and columns of a dense matrix.

[0155] Taking PE0 assigned with index coordinate (0,0) as an example, it can be seen that the row index of PE0 is 0 and the column index is 0. Then, the row vector q0 corresponding to the first row in the query matrix Q is input into PE0 through the first distribution network, and the transposed matrix K corresponding to the key matrix K is T The column vector k0 corresponding to the first column in is input into PE0 through the first distribution network.

[0156] The first calculation matrix in PE0 performs multiplication and accumulation calculation on q0 and k0, and obtains the calculation result 1.5, and outputs the calculation result 1.5 corresponding to PE0. In the specific execution process, the row elements in the row vector q0 corresponding to the first row in the query matrix Q are input into the multiplier one by one through M1, and the transposed matrix K corresponding to the key matrix K is T The column elements in the column vector k0 corresponding to the first column in are input into the multiplier one by one through M2, and the element calculation result corresponding to the first row element and the first column element obtained by the multiplier is 1.2×0.8=0.96, and then the element calculation result corresponding to the second row element and the second column element is 0.5×0.4=0.2, and the third multiplexer M3 is called, and the existing element calculation results and 0.96 are input into the adder, and the element calculation result 0.2 and the existing element calculation results and 0.96 are accumulated according to the adder…that is, 1.2×0.8+0.5×0.4+0.3×0.5+0.1×0.2+0.6×0.3+0.2×0.1=1.56. For the convenience of subsequent calculations, one decimal place is retained here, and the final calculation result s is 1.5, and the calculation result s corresponding to PE0 is output through the fourth multiplexer as 1.5.

[0157] Similarly, the calculation result s corresponding to the output of PE1 is 1.1, the calculation result s corresponding to the output of PE2 is 0.5, and the calculation result s corresponding to the output of PE3 is 0.5. The embodiment of the present application temporarily stores the above calculation result s inside the PE so that it can be reused in the subsequent step S4 when performing the matrix row product calculation, saving the overhead of storing and repeatedly moving the intermediate results.

[0158] It should be noted that the calculation result s corresponding to PE0 is 1.5, the calculation result s corresponding to PE1 is 1.1, the calculation result s corresponding to PE2 is 0.5, and the calculation result s corresponding to PE3 is 0.5, which together constitute the calculation result matrix S. For details, see Figure 5 The calculation result matrix S. It should be noted that this figure is for understanding that the calculation result s in the calculation result matrix S corresponds to the non-zero elements in the sparse matrix block, that is, the calculation result matrix S is also a sparse matrix.

[0159] In a specific implementation process, the embodiment of the present application provides outputting the calculation result through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM, including:

[0160] Determine whether the row element is the last row element in the corresponding row vector in the query matrix;

[0161] If the row element is not the last row element in the row vector corresponding to the query matrix, return to step S31, that is, return to execute the traversal of the row elements in the row vector corresponding to the query matrix and the column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix until the last row element in the row vector corresponding to the query matrix is ​​obtained;

[0162] If the row element is the last row element in the corresponding row vector in the query matrix, the calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0163] In a specific implementation process, the embodiment of the present application provides outputting the calculation result through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM, including:

[0164] Determine whether the column element is the last column element in the corresponding column vector in the transposed key matrix corresponding to the key matrix;

[0165] If the column element is not the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix, then return to execute the traversal of the row elements in the corresponding row vector in the query matrix and the column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix until the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix is ​​obtained;

[0166] If the column element is the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix, the calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0167] It should be noted that two judgment methods are given above to determine whether the calculation of all row elements in the row vector corresponding to the query matrix and all column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix has been completed. Because the number of row elements in the row vector corresponding to the query matrix is ​​equal to the number of column vectors corresponding to the transposed key matrix corresponding to the key matrix, one of the judgment methods can be selected to execute in the specific implementation process.

[0168] In an embodiment of the present application, in the SDDMM mode, the PEs in the one-dimensional systolic array are configured to perform multiplication and accumulation operations. Each row element of the left matrix and each row element of the right matrix (because each column element corresponding to the transposed matrix of the right matrix can be regarded as each row element of the right matrix) continuously flow into the corresponding PE for multiplication and accumulation, and are calculated in the form of matrix inner product to obtain the calculation result s, thereby completing the SDDMM. In this mode, the multiplexers M1 and M2 in the PE respectively transmit element q and element k into the multiplier for multiplication. The multiplexer M3 sends the partial sum to the adder and accumulates it with the new multiplication result until the calculation of the elements in the PE is completed.

[0169] S4, according to the second calculation path in the PE, perform multiplication calculation on the calculation result and the corresponding row vector in the value matrix, and output the multiplication result corresponding to the PE.

[0170] Among them, the corresponding row vector in the value matrix corresponds to the column index of the PE, the calculation result is a part of the calculation result matrix, and the calculation result corresponding to each PE is temporarily stored in the PE.

[0171] It should be noted that the value matrix is ​​usually represented by V. Through the second calculation path of the PE, multiplication calculation is performed on the corresponding row vector in the value matrix V input to the PE, and the multiplication result corresponding to the PE is output.

[0172] In a possible implementation, before S4, the embodiment of the present application may further include:

[0173] Based on the column index of PE, the corresponding row vector in the value matrix is ​​input into PE through the second cross network.

[0174] In the specific implementation process, the value matrix V is a dense matrix, and the value matrix V is a right matrix. Please refer to Figure 4 A schematic diagram of inputting a one-dimensional systolic array through a distribution network is shown. In the embodiment of the present application, based on the column index corresponding to the PE, the corresponding row vector in the value matrix V is input to the PE through the second cross network. Taking PE0 with the assigned index coordinate (0, 0) as an example, the column index of PE0 is 0, then the row vector V0 corresponding to the first row in the value matrix V is input to the PE0 through the second distribution network.

[0175] In a possible implementation, the second computing path includes a first multiplexer, a second multiplexer, a fourth multiplexer, and a multiplier.

[0176] The specific execution process of S4 provided in the embodiment of the present application can be found in Figure 7 and Figure 8 , Figure 7 A schematic diagram of an example of a calculation process of a matrix inner product and a matrix outer product provided in an embodiment of the present application, Figure 8 The schematic diagram of the structure of the second computing path of the reconfigurable PE provided in the embodiment of the present application, S4 may include:

[0177] S41, inputting the calculation result into the multiplier through the first multiplexer.

[0178] S42, inputting the corresponding row vector in the value matrix into the multiplier through the second multiplexer.

[0179] S43, performing multiplication calculation on the calculation result and the corresponding row vector in the value matrix according to the multiplier to obtain the multiplication result.

[0180] S44, outputting the multiplication result through the fourth multiplexer to complete the multiplication of the sparse matrix and the dense matrix (sparse-dense matrix multiplication, SpMM).

[0181] Among them, SpMM refers to multiplying a sparse matrix by a dense matrix to obtain a dense matrix.

[0182] Still taking PE0 assigned with index coordinate (0, 0) as an example, the calculation result s of 1.5 is temporarily stored in PE0, and the calculation result s1.5 is input into the multiplier through the first multiplexer M1. The column index of PE0 is 0, then the row vector v0 corresponding to the first row in the value matrix V is input into the multiplier through the second multiplexer M2.

[0183] The multiplier performs multiplication calculation on the calculation result s and the row vector v0 of the first row corresponding to the value matrix V to obtain the multiplication result. For PE0, 1.5×v0 is calculated and the multiplication result is obtained by using z 0p0 Indicates that z 0p0 =[1.05, 0.75, 0.45, 1.35, 0.60, 0.90], and outputs the z through the fourth multiplexer M4. 0 p 0 .

[0184] Similarly, output the multiplication result z corresponding to PE1 0p2 =[1.65, 2.53, 0.88, 1.32, 2.09, 055], output the multiplication result z corresponding to PE21p1 For z 1p1 =[0.25, 0.40, 0.15, 0.45, 0.20, 0.30], output the multiplication result z corresponding to PE3 2p3 =[0.65, 1.05, 0.45, 0.80, 0.55, 0.35].

[0185] In the embodiment of the present application, in the SpMM mode, the PE in the one-dimensional systolic array is configured to perform a multiplication operation. The sparse elements in the left matrix S are temporarily stored in the PE, and each row of the right matrix continuously flows into the corresponding PE for multiplication. The multiplication result is sent back to the group summation network, and is calculated in the form of a matrix row product to obtain the multiplication result z, thereby completing SpMM. In this mode, the multiplexers M1 and M2 in the PE respectively transmit s and v into the multiplier for multiplication, and the multiplexer M4 directly outputs the multiplication result without accumulating it in the PE.

[0186] In a possible implementation, the row elements in the corresponding row vector in the value matrix may be input into the PE periodically through the second cross network according to the number of cycles, and the number of cycles is equal to the number of columns of the value matrix.

[0187] To facilitate understanding of the solution provided by the embodiment of the present application, the following is explained with reference to PE0 as an example. The number of columns of the value matrix V provided by the embodiment of the present application is 6, so the number of cycles T=6. In the first cycle, the first row element 0.7 in the row vector v0 corresponding to the first row in the value matrix V is input into PE0 through the second cross network. In the second cycle, the second row element 0.5 in the row vector v0 corresponding to the first row in the value matrix V is input into PE0 through the second cross network... In the sixth cycle, the sixth row element 0.6 in the row vector v0 corresponding to the first row in the value matrix V is input into PE0 through the second cross network, thereby completing the input of the corresponding row vector in the value matrix V through the second cross network.

[0188] According to the above steps S3 to S4, the computing scenario of the attention mechanism of the Transfomer model in the embodiment of the present application can convert the original dense matrix multiplication into SDDMM and SpMM, and then the intermediate results of SDDMM and SpMM can be temporarily stored and reused, saving the overhead of storing and repeatedly moving the intermediate results, thereby improving data processing efficiency.

[0189] The data processing method for sparse matrix multiplication calculation provided in the embodiment of the present application is oriented to the sparse attention mechanism of the Tansformer model, and supports the matrix multiplication structure of the matrix inner product (inner product) and the matrix row product (row-wise product). The embodiment of the present application can not only process SDDMM and SpMM, but also is applicable to the matrix multiplication of general matrices (GEneral Matrix to Matrix Multiplication, GEMM), and has high scalability and versatility.

[0190] That is, the embodiment of the present application can support three matrix calculation modes, namely SDDMM, SpMM and GEMM, and has high scalability and versatility. At the same time, when expanding from a single calculation structure to multiple ones, the calculation efficiency and resource usage change linearly, and the expansion cost is low, which makes it possible to process large-scale data and more complex calculation tasks.

[0191] S5, performing group summation on the multiplication results of the number of non-zero elements through a group summation network to obtain at least one group summation result.

[0192] In an embodiment of the present application, it is necessary to group and sum the multiplication results of non-zero elements through a group summation network to obtain at least one group summation result. In the specific implementation process, the multiplication results corresponding to PEs with the same row index are summed.

[0193] For the specific implementation process, please refer to Fig. 9 Schematic diagram of the group summation process and Figure 7 The schematic diagram of an example of the calculation process of the inner product and the outer product of the matrix shown in FIG. 1 is combined with the above example to illustrate that the non-zero element multiplication results, that is, the 4 multiplication results are z 0p0 =[1.05, 0.75, 0.45, 1.35, 0.60, 0.90], z 0p2 =[1.65, 2.53, 0.88, 1.32, 2.09, 055], z 1p1 =[0.25, 0.40, 0.15, 0.45, 0.20, 0.30] and z 2p3 =[0.65, 1.05, 0.45, 0.80, 0.55, 0.35]. z 0p0 The corresponding row index of PE0 is 0, z 0p2 The corresponding row index of PE1 is 0, for z 0p0 and z 0p2 Sum and finally get z 0p0 +z 0p2 =z 0p , z 0p=[2.70,3.28,1.33,2.67,2.69,1.45]. It should be noted that, because the other PE row indexes are not the same, at least one group summation result is finally obtained including z 0p 、z 1p and z 2p .

[0194] In the specific implementation process, the group summation network includes registers. Although the registers exist in a tree topology, the number of registers at each level in the tree topology is the same as the preset number of PEs in the one-dimensional systolic array, thereby completing the distinction and summation of elements of different groups.

[0195] It should be noted that the group summation network is designed to support the calculation mode of matrix row product. Matrix row product requires that the non-zero elements of a row of the calculation result matrix S (left matrix) be multiplied one by one with the elements in the corresponding row vector of the value matrix V (right matrix), and the multiple rows obtained are then accumulated at the corresponding positions. Therefore, the group summation network determines whether it is a type of row that needs to be summed according to the row index saved in PE. After that, each result of the same type of row is accumulated in the group summation network after being generated in PE, and is no longer accumulated in PE, thereby completing the summation of the intermediate result rows of the matrix row product.

[0196] In the embodiment of the present application, the group summation network performs group summation according to the row index in the PE, and completes the summation of the intermediate results of the matrix row product. The group summation network can accurately sum the elements of the same row, avoiding data confusion and erroneous calculations, and improving the accuracy and efficiency of the calculation.

[0197] S6, reorder at least one group summation result according to the row index corresponding to the group summation result to obtain an output matrix.

[0198] Among them, the output matrix corresponds to the sparse matrix block.

[0199] In the embodiment of the present application, because the at least one group summation result obtained in step S5 includes z 0p 、z 1p and z 2p , at least one group summation result is not necessarily arranged in the order of arrangement, so it needs to be reordered to finally obtain the output matrix Z, which is the result obtained after the corresponding data processing of the sparse matrix block. The reordering makes the row index the same as the natural sequence number of the array.

[0200] In the specific implementation process, due to the step-by-step storage characteristics of the group summation network, each sum of its final output is distributed at any position where the corresponding group element is located, so it is also necessary to reorder each sum according to the row index so that its row index is the same as the natural sequence number of the array.

[0201] It should be noted that through the above series of designs, the accelerator can maintain a high hardware utilization rate at different sparsity levels. When the sparsity is high, the hardware efficiency of the embodiment of the present application is higher than that of other methods in the prior art. Because the computing unit PE can be fully utilized, the idle time is reduced and the output processing efficiency of the entire accelerator is improved. For specific comparison results, please refer to Fig.10 , Fig.10 A comparison chart of computing efficiency under different sparsities provided for an embodiment of the present application shows that the embodiment of the present application can still maintain high utilization and computing efficiency under higher sparsity conditions, and has a 20% efficiency improvement compared to the traditional two-dimensional systolic array solution.

[0202] Thus, the method provided in the embodiment of the present application can uniformly support the calculation of two sparse modes, SDDMM and SpMM, and make full use of the characteristics of matrix inner product and matrix row product to efficiently process the two types of sparse matrix multiplication, SDDMM and SpMM. T Corresponding to SDDMM, S×V corresponds to SpMM. By connecting the two, the calculation result s in the intermediate calculation result matrix S can be temporarily stored in the corresponding PE. After both modes are calculated, the calculation result s in PE can be directly discarded, reducing the storage and data exchange overhead.

[0203] See also Fig.11 , the embodiment of the present application further provides a data processing device 1100 for sparse matrix multiplication calculation, the device 1100 comprising: a sparse index preprocessing module 1101, a distribution network module 1102, a one-dimensional systolic array module 1103 and a group summation network module 1104;

[0204] The sparse index preprocessing module 1101 is used to preprocess the sparse matrix to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; the sparse index includes the number of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements;

[0205] The distribution network module 1102 is used to allocate the corresponding index coordinates to the computing units PE in the one-dimensional systolic array according to the number of non-zero elements in the sparse index; the index coordinates include a row index and a column index; the one-dimensional systolic array includes a preset number of PEs arranged in parallel; the PE is a reconfigurable PE including a first computing path and a second computing path; the first computing path is used to perform a matrix inner product operation, and the second computing path is used to perform a matrix row product operation;

[0206] The one-dimensional systolic array module 1103 is used to perform multiplication and accumulation calculations on the corresponding row vectors in the query matrix and the corresponding column vectors in the transposed key matrix corresponding to the key matrix according to the first calculation path in the PE, and output the calculation results corresponding to the PE; the corresponding row vectors in the query matrix correspond to the row index of the PE, and the corresponding column vectors in the transposed key matrix corresponding to the key matrix correspond to the column index of the PE;

[0207] The one-dimensional systolic array module 1103 is further used to perform a multiplication calculation on the calculation result and a corresponding row vector in a value matrix according to the second calculation path in the PE, and output a multiplication result corresponding to the PE; the corresponding row vector in the value matrix corresponds to the column index of the PE;

[0208] The group summation network module 1104 is used to perform group summation on the multiplication results of the number of non-zero elements through a group summation network to obtain at least one group summation result;

[0209] The group summation network module 1104 is further used to reorder at least one of the group summation results according to the row index corresponding to the group summation result to obtain an output matrix; the output matrix corresponds to the sparse matrix block.

[0210] Optionally, the first computing path includes a first multiplexer, a second multiplexer, a third multiplexer, a fourth multiplexer, a multiplier, and an adder, and the one-dimensional systolic array module 1103 is specifically used for:

[0211] Traversing the row elements in the corresponding row vector in the query matrix and the column elements in the corresponding column vector in the transposed key matrix corresponding to the key matrix; the column index of the row element corresponds to the row index of the column element;

[0212] Input the row elements into the multiplier through the first multiplexer;

[0213] Input the column elements into the multiplier through the second multiplexer;

[0214] Performing multiplication calculation on the row element and the row element according to the multiplier to obtain an element calculation result;

[0215] Calling the third multiplexer to input the existing element calculation result and the sum into the adder; the existing element calculation result and the sum of the element calculation results that have been accumulated and calculated by the adder;

[0216] Accumulate the element calculation result and the existing element calculation result according to the adder to obtain the calculation result;

[0217] The calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0218] Optionally, the second computing path includes a first multiplexer, a second multiplexer, a fourth multiplexer and a multiplier, and the one-dimensional systolic array module 1103 is specifically used for:

[0219] Inputting the calculation result into the multiplier through the first multiplexer;

[0220] Inputting the corresponding row vector in the value matrix into the multiplier through the second multiplexer;

[0221] Performing a multiplication calculation on the calculation result and a corresponding row vector in the value matrix according to the multiplier to obtain the multiplication result;

[0222] The multiplication result is output through the fourth multiplexer to complete the multiplication SpMM of the sparse matrix and the dense matrix.

[0223] Optionally, the one-dimensional systolic array module 1103 is specifically used for:

[0224] Determine whether the row element is the last row element in the corresponding row vector in the query matrix;

[0225] If the row element is not the last row element in the row vector corresponding to the query matrix, then return to execute the traversal of the row elements in the row vector corresponding to the query matrix and the column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix until the last row element in the row vector corresponding to the query matrix is ​​obtained;

[0226] If the row element is the last row element in the corresponding row vector in the query matrix, the calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0227] Optionally, the one-dimensional systolic array module 1103 is specifically used for:

[0228] Determine whether the column element is the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix;

[0229] If the column element is not the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix, then returning to execute the traversal of the row elements in the corresponding row vector in the query matrix and the column elements in the column vector corresponding to the transposed key matrix corresponding to the key matrix until the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix is ​​obtained;

[0230] If the column element is the last column element in the column vector corresponding to the transposed key matrix corresponding to the key matrix, the calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

[0231] Optionally, the distribution network module 1102 is further used to:

[0232] Based on the row index of the PE, input the corresponding row vector in the query matrix into the PE through the first cross network;

[0233] Based on the column index of the PE, a column vector corresponding to the transposed key matrix corresponding to the key matrix is ​​input into the PE through the first cross network.

[0234] Optionally, the distribution network module 1102 is further used to:

[0235] Based on the column index of the PE, the corresponding row vector in the value matrix is ​​input into the PE through a second cross network.

[0236] Optionally, the distribution network module 1102 is specifically configured to:

[0237] The row elements in the corresponding row vector in the value matrix are input into the PE through the second cross network cycle by cycle according to the number of cycles, and the number of cycles is equal to the number of columns of the value matrix.

[0238] The sparse index preprocessing module 1101 is specifically used for:

[0239] Divide the sparse matrix into blocks to obtain the sparse matrix blocks;

[0240] Obtaining the number of non-zero elements and the index coordinates of the non-zero elements in the sparse matrix block;

[0241] The number of non-zero elements and the index coordinates are arranged in one dimension to obtain the sparse index corresponding to the sparse matrix block.

[0242] Thus, the embodiment of the present application provides a data processing device for sparse matrix multiplication. In the specific implementation process, the device can be regarded as a computing structure, which consists of a sparse index preprocessing module, a distribution network module, a one-dimensional systolic array module and a group summation network module. For the input sparse matrix, the sparse data is first arranged into a tightly connected one-dimensional format through the sparse index preprocessing module, and then part of the sparse data is sent to the one-dimensional systolic array module for calculation through the distribution network module. The one-dimensional systolic array consists of multiple multiplication and accumulators. When performing matrix inner product calculation, the results of the multipliers are accumulated in the array; when performing matrix row product calculation, the results of the multipliers are not accumulated in the array, but directly sent to the group summation network module for summation. The matrix inner product is suitable for the SDDMM operation of the attention mechanism in the Transformer model, while the matrix row product is suitable for the SpMM operation therein. That is, the solution provided by the embodiment of the present application supports the processing of both at the same time, which can save hardware resources and reduce the storage and handling overhead of intermediate results. At the same time, the computing structure also supports GEMM operation, which has high scalability and versatility.

[0243] It should be noted that the specific implementation of the device 1100 can be found in Fig.12 The overall process execution diagram of a data processing device 1100 for sparse matrix multiplication calculation is shown in FIG. The technical effect achieved by the device 1100 can be seen in FIG. Figure 1 The relevant description in the method shown.

[0244] Among them, for Fig.12 The matrix block part and the summation module are explained in the figure. Because for each sparse matrix block, the number of non-zero elements in it may not be less than or equal to the preset data of PE in the one-dimensional systolic array, it is necessary to sum the output matrix of each calculation to finally obtain the output matrix Z corresponding to the sparse matrix block.

[0245] For ease of understanding, the following two situations are explained:

[0246] In the embodiment of the present application, there are 4 non-zero elements in the sparse matrix block A, and their number is equal to the number of PEs in the one-dimensional systolic matrix. The result after grouping and summing is: 0p =[2.70,3.28,1.33,2.67,2.69,1.45], z 1p =[0.25, 0.40, 0.15, 0.45, 0.20, 0.30] and z 2p =[0.65, 1.05, 0.45, 0.80, 0.55, 0.35] is the final output result.

[0247] However, for the 5 non-zero elements in the sparse matrix block B, there are only 4 PEs in the one-dimensional systolic array, so the calculation cannot be completed at one time. Then, the output matrix Z corresponding to (0, 1), (1, 2), (2, 1), and (2, 3) is calculated first. 1 , and then calculate the output matrix Z corresponding to (3, 2) 2 , through the matrix block part and summation module, Z 1 and Z 2 Sum and finally get the output matrix Z.

[0248] In addition, the embodiment of the present application also provides an electronic device 1300, such as Fig.13 As shown, the electronic device 1300 includes a processor 1301 and a memory 1302:

[0249] The memory 1302 is used to store computer programs;

[0250] The processor 1301 is used to execute according to the computer program Figure 1 The method provided.

[0251] It should be noted that, in the actual implementation process, the processor 1301 can be a processor in the Transformer model or an accelerator in the Transformer model.

[0252] In addition, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method provided by the embodiment of the present application.

[0253] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0254] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, in which the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the goals of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0255] The above is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. It should be noted that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the present application, and these improvements and modifications should also be regarded as the protection scope of the present application.

Claims

1. A data processing method for sparse matrix multiplication calculation, characterized in that: The method comprises: Preprocessing the sparse matrix to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; the sparse index includes the number of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements; According to the number of non-zero elements in the sparse index, the corresponding index coordinates are allocated to computing units PE in a one-dimensional systolic array; the index coordinates include a row index and a column index; the one-dimensional systolic array includes a preset number of PEs arranged in parallel; the PE is a reconfigurable PE including a first computing path and a second computing path; the first computing path is used to perform a matrix inner product operation, and the second computing path is used to perform a matrix row product operation; According to the first calculation path in the PE, perform multiplication and accumulation calculations on the corresponding row vectors in the query matrix and the corresponding column vectors in the transposed key matrix corresponding to the key matrix, and output the calculation results corresponding to the PE; the corresponding row vectors in the query matrix correspond to the row index of the PE, and the corresponding column vectors in the transposed key matrix corresponding to the key matrix correspond to the column index of the PE; According to the second calculation path in the PE, perform a multiplication calculation on the calculation result and a corresponding row vector in a value matrix, and output a multiplication result corresponding to the PE; the corresponding row vector in the value matrix corresponds to the column index of the PE; Performing group summation on the multiplication results of the number of non-zero elements through a group summation network to obtain at least one group summation result; At least one of the grouped summation results is reordered according to the row index corresponding to the grouped summation results to obtain an output matrix; the output matrix corresponds to the sparse matrix block.

2. The method according to claim 1, characterized in that The first calculation path includes a first multiplexer, a second multiplexer, a third multiplexer, a fourth multiplexer, a multiplier, and an adder. The multiplication and accumulation calculation is performed on the corresponding row vector in the query matrix and the corresponding column vector in the transposed key matrix corresponding to the key matrix according to the first calculation path in the PE, and the calculation result corresponding to the PE is output, including: Traversing the row elements in the corresponding row vector in the query matrix and the column elements in the corresponding column vector in the transposed key matrix corresponding to the key matrix; the column index of the row element corresponds to the row index of the column element; Input the row elements into the multiplier through the first multiplexer; Input the column elements into the multiplier through the second multiplexer; Performing multiplication calculation on the row element and the row element according to the multiplier to obtain an element calculation result; Calling the third multiplexer to input the existing element calculation result and the sum into the adder; the existing element calculation result and the sum of the element calculation results that have been accumulated and calculated by the adder; Accumulate the element calculation result and the existing element calculation result according to the adder to obtain the calculation result; The calculation result is output through the fourth multiplexer to complete the sampled dense-dense matrix multiplication SDDMM.

3. The method according to claim 1, characterized in that The second calculation path includes a first multiplexer, a second multiplexer, a fourth multiplexer and a multiplier, and the multiplication calculation is performed on the calculation result and the corresponding row vector in the value matrix according to the second calculation path in the PE, and the multiplication result corresponding to the PE is output, including: Inputting the calculation result into the multiplier through the first multiplexer; Inputting the corresponding row vector in the value matrix into the multiplier through the second multiplexer; Performing a multiplication calculation on the calculation result and a corresponding row vector in the value matrix according to the multiplier to obtain the multiplication result; The multiplication result is output through the fourth multiplexer to complete the multiplication SpMM of the sparse matrix and the dense matrix.

4. The method according to claim 1, characterized in that Before performing multiplication and accumulation calculations on the corresponding row vectors in the query matrix and the corresponding column vectors in the transposed key matrix corresponding to the key matrix according to the first calculation path in the PE and outputting the calculation results corresponding to the PE, the method further includes: Based on the row index of the PE, input the corresponding row vector in the query matrix into the PE through the first cross network; Based on the column index of the PE, a column vector corresponding to the transposed key matrix corresponding to the key matrix is ​​input into the PE through the first cross network.

5. The method according to claim 1, characterized in that Before performing a multiplication calculation on the calculation result and a corresponding row vector in the value matrix according to the second calculation path in the PE and outputting the multiplication result corresponding to the PE, the method further includes: Based on the column index of the PE, the corresponding row vector in the value matrix is ​​input into the PE through a second cross network.

6. The method according to claim 5, characterized in that The step of inputting the corresponding row vector in the value matrix into the PE through a second cross network comprises: The row elements in the corresponding row vector in the value matrix are input into the PE through the second cross network cycle by cycle according to the number of cycles, and the number of cycles is equal to the number of columns of the value matrix.

7. The method according to claim 1, characterized in that The preprocessing of the sparse matrix to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block includes: Divide the sparse matrix into blocks to obtain the sparse matrix blocks; Obtaining the number of non-zero elements and the index coordinates of the non-zero elements in the sparse matrix block; The number of non-zero elements and the index coordinates are arranged in one dimension to obtain the sparse index corresponding to the sparse matrix block.

8. A data processing device for sparse matrix multiplication calculation, characterized in that: The device comprises: a sparse index preprocessing module, a distribution network module, a one-dimensional systolic array module and a group summation network module; The sparse index preprocessing module is used to preprocess the sparse matrix to obtain a sparse matrix block and a sparse index corresponding to the sparse matrix block; the sparse index includes the number of non-zero elements in the sparse matrix block and the index coordinates of the non-zero elements; The distribution network module is used to allocate the corresponding index coordinates to the computing units PE in the one-dimensional systolic array according to the number of non-zero elements in the sparse index; the index coordinates include a row index and a column index; the one-dimensional systolic array includes a preset number of PEs arranged in parallel; the PE is a reconfigurable PE including a first computing path and a second computing path; the first computing path is used to perform a matrix inner product operation, and the second computing path is used to perform a matrix row product operation; The one-dimensional systolic array module is used to perform multiplication and accumulation calculations on the corresponding row vectors in the query matrix and the corresponding column vectors in the transposed key matrix corresponding to the key matrix according to the first calculation path in the PE, and output the calculation results corresponding to the PE; the corresponding row vectors in the query matrix correspond to the row index of the PE, and the corresponding column vectors in the transposed key matrix corresponding to the key matrix correspond to the column index of the PE; The one-dimensional systolic array module is further used to perform a multiplication calculation on the calculation result and a corresponding row vector in a value matrix according to the second calculation path in the PE, and output a multiplication result corresponding to the PE; the corresponding row vector in the value matrix corresponds to the column index of the PE; The group summation network module is used to perform group summation on the multiplication results of the number of non-zero elements through a group summation network to obtain at least one group summation result; The group summation network module is further used to reorder at least one of the group summation results according to the row index corresponding to the group summation result to obtain an output matrix; the output matrix corresponds to the sparse matrix block.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory: The memory is used to store computer programs; The processor is configured to execute the method according to any one of claims 1 to 7 according to the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method according to any one of claims 1 to 7.

Images