Universal matrix multiplication operation device and method and electronic equipment

Abstract

The invention relates to the technical field of computers, and provides a general matrix multiplication operation device and method and electronic equipment. In the device, a first storage module stores a first data matrix and a second data matrix, the column number of the first data matrix is s, and the row number of the second data matrix is s; the mask generation module generates a composite mask according to the first data matrix and the second data matrix; the vector generation module generates a first vector and a second vector according to the composite mask, the first data matrix and the second data matrix, the number of elements of the first vector is equal to the number of elements of the second vector and is smaller than s, and the operation module calculates the product of the first vector and the second vector, the product is equal to the product of the ith row element of the first data matrix and the jth column element of the second data matrix; the second storage module stores a product of the first vector and the second vector. The GEMM algorithm realized by the device can reduce the amount of data to be calculated, so that the calculation power consumption of the device is reduced, and the energy efficiency of the device is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a general matrix multiplication operation device, method, and electronic device. Background Art

[0002] General Matrix Multiplication (GEMM) is an algorithm frequently used in the field of computer technology. It can efficiently perform matrix multiplication operations and is widely used in various scientific computing, graphics processing, and artificial intelligence fields. However, matrix multiplication operations are very complex, and how to implement an efficient GEMM algorithm has always been a research hotspot in this field.

[0003] The existing technology mainly optimizes the computing performance and bandwidth bottleneck of hardware devices supporting the GEMM algorithm (such as graphics processing units (GPUs), artificial intelligence (AI) accelerators, etc.). The overall computational workload of the GEMM algorithm has not changed, so the computing power consumption of the device is large and the energy efficiency of the device is low. How to reduce the computing power consumption of the device and improve the energy efficiency of the device has become a technical problem that needs to be solved urgently in this field. Summary of the invention

[0004] In view of this, the present disclosure proposes a general matrix multiplication operation device, method, and electronic device. The GEMM algorithm implemented by the general matrix multiplication operation device of the present disclosure can reduce the amount of data to be calculated, thereby reducing the computing power consumption of the device and improving the energy efficiency of the device.

[0005] According to one aspect of the present disclosure, a general matrix multiplication device is provided, the device being used to perform a general matrix multiplication operation on a first data matrix and a second data matrix to obtain a third data matrix, wherein the first data matrix has m rows and s columns, the second data matrix has s rows and n columns, and the third data matrix has m rows and n columns, where m, s, and n are positive integers, and the device comprises: a first storage module for storing the first data matrix and the second data matrix; a mask generation module for generating a composite mask corresponding to the i-th row element of the first data matrix and the j-th column element of the second data matrix according to the first data matrix and the second data matrix; and a vector generation module for generating a vector according to the composite mask. A first vector is generated according to the composite mask and the i-th row elements of the first data matrix, and a second vector is generated according to the composite mask and the j-th column elements of the second data matrix, the number of elements of the first vector is equal to the number of elements of the second vector and is less than s, i≤m, j≤n, i and j are positive integers; an operation module is used to calculate the product of the first vector and the second vector, and the product is equal to the product of the i-th row elements of the first data matrix and the j-th column elements of the second data matrix; a second storage module is used to store the third data matrix, when the general matrix multiplication operation of the first data matrix and the second data matrix is ​​completed, the elements of the i-th row and the j-th column in the third data matrix are the products of the first vector and the second vector.

[0006] In a possible implementation, the mask generation module includes: a mask matrix generation unit, used to generate a first mask matrix according to the first data matrix, and to generate a second mask matrix according to the second data matrix; a first mask storage unit, used to store the first mask matrix; a second mask storage unit, used to store the second mask matrix; a composite mask generation unit, used to generate the composite mask according to the i-th row elements of the first mask matrix and the j-th column elements of the second mask matrix; wherein the first mask matrix has m rows and j columns. s, the number of rows and columns of the second mask matrix is ​​s and n, the element in the i-th row and k-th column of the first mask matrix indicates whether the element in the i-th row and k-th column of the first data matrix is ​​0, the element in the k-th row and j-th column of the second mask matrix indicates whether the element in the k-th row and j-th column of the second data matrix is ​​0, k≤s, k is a positive integer; the composite mask includes s elements, and the k-th element of the composite mask indicates whether the product of the element in the i-th row and k-th column of the first data matrix and the element in the k-th row and j-th column of the second data matrix is ​​0.

[0007] In a possible implementation, when the element in the i-th row and k-th column of the first mask matrix is ​​0, it indicates that the element in the i-th row and k-th column of the first data matrix is ​​also 0; when the element in the i-th row and k-th column of the first mask matrix is ​​1, it indicates that the element in the i-th row and k-th column of the first data matrix is ​​not 0; when the element in the k-th row and j-th column of the second mask matrix is ​​0, it indicates that the element in the k-th row and j-th column of the second data matrix is ​​also 0; when the element in the k-th row and j-th column of the second mask matrix is ​​1, it indicates that the element in the k-th row and j-th column of the second data matrix is ​​not 0.

[0008] In a possible implementation, the composite mask generation unit is specifically used to perform an AND operation on the elements of the i-th row of the first mask matrix and the elements of the j-th column of the second mask matrix to obtain the composite mask; when the k-th element of the composite mask is 0, it indicates that the product of the elements of the i-th row and k-th column in the first data matrix and the elements of the k-th row and j-th column in the second data matrix is ​​0; when the k-th element of the composite mask is 1, it indicates that the product of the elements of the i-th row and k-th column in the first data matrix and the elements of the k-th row and j-th column in the second data matrix is ​​not 0.

[0009] In a possible implementation, the vector generation module is specifically used to perform an AND operation on the composite mask and the i-th row elements of the first data matrix, remove the elements with a value of 0 in the operation result and densely arrange the remaining elements to obtain the first vector; perform a logical AND operation on the composite mask and the j-th column elements of the second data matrix, remove the elements with a value of 0 in the operation result and densely arrange the remaining elements to obtain the second vector.

[0010] In a possible implementation, the device further includes a controller, the operation module completes a plurality of multiplication operations of vectors with a number of elements of w in one calculation cycle, where w is a positive integer less than s; the controller is used to split the first vector and the second vector according to the value of w, the first vector is split into z first sub-vectors, the second vector is split into z second sub-vectors, the t-th first sub-vector and the t-th second sub-vector include the same number of elements, t≤z, t and z are positive integers; the number of elements of the 1st to z-1st first sub-vectors is The number is equal to w, and the number of elements of the z-th first sub-vector is less than or equal to w; the operation module is specifically used to calculate the product of the first vector and the second vector through z calculation cycles, and in the t-th calculation cycle, write the calculation result of the t-th calculation cycle into the i-th row and j-th column of the third data matrix, the calculation result of the t-th calculation cycle is equal to the sum of the product of the t-th first sub-vector and the t-th second sub-vector and the calculation result of the t-1-th calculation cycle, and the calculation result of the z-th calculation cycle is the product of the first vector and the second vector.

[0011] In a possible implementation, in the tth calculation cycle, the controller is further used to control the vector generation module to output the tth first sub-vector and the tth second sub-vector to the operation module; when t is greater than 1, the controller is also used to control the second storage module to output the calculation result of the t-1th calculation cycle to the operation module.

[0012] In a possible implementation, the operation module includes multiple operation units, each operation unit completes a multiplication operation of a pair of vectors with w elements in one calculation cycle, and when any operation unit is used to calculate the product of the first vector and the second vector, the operation unit is specifically used to: in the first calculation cycle, calculate the product of the first first sub-vector and the first second sub-vector, as the calculation result of the first calculation cycle, and write the calculation result of the first calculation cycle into the i-th row and j-th column of the third data matrix; when t is greater than 1, in the t-th calculation cycle, calculate the product of the t-th first sub-vector and the t-th second sub-vector, and use the sum of the product and the calculation result of the t-1-th calculation cycle as the calculation result of the t-th calculation cycle, and write the calculation result of the t-th calculation cycle into the i-th row and j-th column of the third data matrix.

[0013] According to another aspect of the present disclosure, a general matrix multiplication method is provided, and the method is applied to a general matrix multiplication device, and the device is used to perform a general matrix multiplication operation on a first data matrix and a second data matrix to obtain a third data matrix, wherein the first data matrix has m rows and s columns, the second data matrix has s rows and n columns, and the third data matrix has m rows and n columns, and m, s, and n are positive integers. The device includes a first storage module, a mask generation module, a vector generation module, an operation module, and a second storage module, and the method includes: using the first storage module to store the first data matrix and the second data matrix; using the mask generation module to generate, according to the first data matrix and the second data matrix, a matrix corresponding to the i-th row element of the first data matrix and the i-th row element of the second data matrix A composite mask corresponding to the j-th column elements; using the vector generation module to generate a first vector according to the composite mask and the i-th row elements of the first data matrix, and generating a second vector according to the composite mask and the j-th column elements of the second data matrix, the number of elements of the first vector is equal to the number of elements of the second vector and is less than s, i≤m, j≤n, i and j are positive integers; using the operation module to calculate the product of the first vector and the second vector, which product is equal to the product of the i-th row elements of the first data matrix and the j-th column elements of the second data matrix; using the second storage module to store the third data matrix, when the general matrix multiplication operation of the first data matrix and the second data matrix is ​​completed, the element of the i-th row and j-th column in the third data matrix is ​​the product of the first vector and the second vector.

[0014] In a possible implementation, the mask generation module includes a mask matrix generation unit, a first mask storage unit, a second mask storage unit, and a composite mask generation unit, wherein the generation of a composite mask corresponding to the i-th row element of the first data matrix and the j-th column element of the second data matrix according to the first data matrix and the second data matrix includes: using the mask matrix generation unit to generate a first mask matrix according to the first data matrix, and generating a second mask matrix according to the second data matrix; using the first mask storage unit to store the first mask matrix; using the second mask storage unit to store the second mask matrix; using the composite mask generation unit to generate a composite mask corresponding to the i-th row element of the first data matrix and the j-th column element of the second data matrix; The i-th row elements of the matrix and the j-th column elements of the second mask matrix generate the composite mask; wherein the first mask matrix has m rows and s columns, the second mask matrix has s rows and n columns, the elements in the i-th row and k-th column of the first mask matrix indicate whether the elements in the i-th row and k-th column of the first data matrix are 0, and the elements in the k-th row and j-th column of the second mask matrix indicate whether the elements in the k-th row and j-th column of the second data matrix are 0, k≤s, k is a positive integer; the composite mask includes s elements, and the k-th element of the composite mask indicates whether the product of the elements in the i-th row and k-th column of the first data matrix and the elements in the k-th row and j-th column of the second data matrix is ​​0.

[0015] In a possible implementation, when the element in the i-th row and k-th column of the first mask matrix is ​​0, it indicates that the element in the i-th row and k-th column of the first data matrix is ​​also 0; when the element in the i-th row and k-th column of the first mask matrix is ​​1, it indicates that the element in the i-th row and k-th column of the first data matrix is ​​not 0; when the element in the k-th row and j-th column of the second mask matrix is ​​0, it indicates that the element in the k-th row and j-th column of the second data matrix is ​​also 0; when the element in the k-th row and j-th column of the second mask matrix is ​​1, it indicates that the element in the k-th row and j-th column of the second data matrix is ​​not 0.

[0016] In a possible implementation, the generating the composite mask according to the i-th row elements of the first mask matrix and the j-th column elements of the second mask matrix includes: performing an AND operation on the i-th row elements of the first mask matrix and the j-th column elements of the second mask matrix to obtain the composite mask; when the k-th element of the composite mask is 0, it indicates that the product of the elements in the i-th row and k-th column of the first data matrix and the elements in the k-th row and j-th column of the second data matrix is ​​0; when the k-th element of the composite mask is 1, it indicates that the product of the elements in the i-th row and k-th column of the first data matrix and the elements in the k-th row and j-th column of the second data matrix is ​​not 0.

[0017] In a possible implementation, generating a first vector according to the composite mask and the i-th row elements of the first data matrix, and generating a second vector according to the composite mask and the j-th column elements of the second data matrix, includes: performing an AND operation on the composite mask and the i-th row elements of the first data matrix, removing elements with a value of 0 in the operation result and densely arranging the remaining elements to obtain the first vector; performing a logical AND operation on the composite mask and the j-th column elements of the second data matrix, removing elements with a value of 0 in the operation result and densely arranging the remaining elements to obtain the second vector.

[0018] In a possible implementation, the device further includes a controller, the operation module completes a plurality of multiplication operations of vectors having a number of elements of w in one calculation cycle, where w is a positive integer less than s, and the method further includes: using the controller to split the first vector and the second vector according to the value of w, the first vector is split into z first sub-vectors, the second vector is split into z second sub-vectors, the t-th first sub-vector and the t-th second sub-vector include the same number of elements, t≤z, t and z are positive integers; the number of elements of the 1st to z-1th first sub-vectors is equal to w, the number of elements of the zth first sub-vector is less than or equal to w; the calculating the product of the first vector and the second vector includes: calculating the product of the first vector and the second vector through z calculation cycles, and in the tth calculation cycle, writing the calculation result of the tth calculation cycle into the i-th row and j-th column of the third data matrix, the calculation result of the tth calculation cycle is equal to the sum of the product of the tth first sub-vector and the tth second sub-vector and the calculation result of the t-1th calculation cycle, and the calculation result of the zth calculation cycle is the product of the first vector and the second vector.

[0019] In a possible implementation, the method further includes: in the tth calculation cycle, using the controller to control the vector generation module to output the tth first sub-vector and the tth second sub-vector to the operation module; when t is greater than 1, using the controller to control the second storage module to output the calculation result of the t-1th calculation cycle to the operation module.

[0020] In a possible implementation, the operation module includes multiple operation units, each operation unit completes a multiplication operation of a pair of vectors with w elements in one calculation cycle, and the product of the first vector and the second vector is calculated through z calculation cycles, including: when any operation unit is used to calculate the product of the first vector and the second vector, in the first calculation cycle, the operation unit is used to calculate the product of the first first sub-vector and the first second sub-vector, as the calculation result of the first calculation cycle, and the calculation result of the first calculation cycle is written into the i-th row and j-th column of the third data matrix; when t is greater than 1, in the t-th calculation cycle, the operation unit is used to calculate the product of the t-th first sub-vector and the t-th second sub-vector, the sum of the product and the calculation result of the t-1-th calculation cycle is used as the calculation result of the t-th calculation cycle, and the calculation result of the t-th calculation cycle is written into the i-th row and j-th column of the third data matrix.

[0021] According to another aspect of the present disclosure, an electronic device is provided, comprising the general matrix multiplication operation device described above.

[0022] According to the general matrix multiplication operation device of the embodiment of the present disclosure, the first storage module stores the first data matrix and the second data matrix, the mask generation module generates a composite mask corresponding to the i-th row elements of the first data matrix and the j-th column elements of the second data matrix according to the first data matrix and the second data matrix, the vector generation module generates a first vector according to the composite mask and the i-th row elements of the first data matrix, and generates a second vector according to the composite mask and the j-th column elements of the second data matrix, the number of elements of the first vector is equal to the number of elements of the second vector and is less than s, i≤m, j≤n, i and j are positive integers, so that the sparsity of the first vector is lower than the sparsity of the i-th row elements of the first data matrix, and the sparsity of the second vector is lower than the sparsity of the j-th column elements of the second data matrix; the operation module calculates the product of the first vector and the second vector, and the product is equal to the product of the i-th row elements of the first data matrix and the j-th column elements of the second data matrix, so that accurate calculation results can be obtained with less calculated data. The second storage module stores a third data matrix, and when the general matrix multiplication operation of the first data matrix and the second data matrix is ​​completed, the element of the i-th row and the j-th column in the third data matrix is ​​the product of the first vector and the second vector, so that the third data matrix can be used as the operation result of the general matrix multiplication operation of the first data matrix and the second data matrix. The GEMM algorithm implemented by the general matrix multiplication operation device of the present disclosure takes into account the sparsity of the elements of each matrix multiplied, which can reduce the amount of data to be calculated, thereby reducing the computing power consumption of the device and improving the energy efficiency of the device.

[0023] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0025] Figure 1 A schematic diagram showing the structure of a prior art GEMM device.

[0026] Figure 2a An example of a calculation performed by a prior art operation array over multiple calculation cycles is shown.

[0027] Figure 2b An example of a calculation performed by a prior art operation array over multiple calculation cycles is shown.

[0028] Figure 3 An example of calculation completed in multiple calculation cycles by a single calculation unit in a calculation array of the prior art is shown.

[0029] Figure 4 An exemplary application scenario of the general matrix multiplication operation device according to an embodiment of the present disclosure is shown.

[0030] Figure 5 A schematic diagram showing the structure of a general matrix multiplication operation device according to an embodiment of the present disclosure.

[0031] Figure 6 A schematic diagram showing the structure of a mask generation module according to an embodiment of the present disclosure.

[0032] Figure 7 Examples of elements in a first data matrix, a second data matrix, a first mask matrix, a second mask matrix, and a composite mask according to an embodiment of the present disclosure are shown.

[0033] Figure 8 Examples of elements in a first data matrix, a second data matrix, a composite mask, a first vector, and a second vector according to an embodiment of the present disclosure are shown.

[0034] Fig. 9 A schematic diagram showing the structure of a general matrix multiplication operation device according to an embodiment of the present disclosure.

[0035] Fig.10 A schematic diagram showing the structure of a computing unit according to an embodiment of the present disclosure.

[0036] Fig.11 An example is shown in which a calculation unit according to an embodiment of the present disclosure calculates the product of a first vector and a second vector through multiple calculation cycles.

[0037] Fig.12A schematic diagram showing the process of a general matrix multiplication method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0039] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0040] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0041] The following are terms that may appear in this article.

[0042] Sparsity: refers to the proportion of elements with a value of 0 in the data. The more elements with a value of 0 in the data, the higher the sparsity. Conversely, the fewer elements with a value of 0, the lower the sparsity.

[0043] Sparse data: data that includes at least one element with a value of 0.

[0044] Dense data: data in which all elements are not zero.

[0045] Energy efficiency: The computing power provided per unit power consumption.

[0046] The following describes how the GEMM device in the prior art implements the GEMM algorithm.

[0047] Figure 1 A schematic diagram showing the structure of a prior art GEMM device.

[0048] like Figure 1 As shown, the GEMM device of the prior art mainly includes a first cache, a second cache, an operation array, a controller and a third cache. The first cache and the second cache are respectively used to store the matrix A and the matrix B to be operated, the operation array is used to complete the general matrix multiplication operation of the matrix A and the matrix B through multiple calculation cycles under the control of the controller, and the third cache is used to store the general matrix multiplication operation result of the matrix A and the matrix B, that is, the matrix D.

[0049] Among them, matrix A and matrix B will be split into multiple small matrices, and the product of the small matrix of matrix A and the small matrix of matrix B will be calculated in each cycle. The calculation results of each cycle are cyclically accumulated to obtain the final calculation result.

[0050] Figure 2a and Figure 2b An example of a calculation performed by a prior art operation array over multiple calculation cycles is shown.

[0051] like Figure 2a As shown, both matrix A and matrix B are matrices with 32 rows and 32 columns of elements, and the operation array includes 64 operation units, each of which can complete the multiplication operation of a vector with 1 row and 8 columns of elements and a vector with 8 rows and 1 column of elements in one calculation cycle, and optionally, can also complete an addition operation of elements and elements. In other words, the operation array can complete the multiplication operation of 2 matrices with 8 rows and 8 columns of elements in one calculation cycle.

[0052] In this case, we can first split matrix A into 4 matrices with 8 rows and 32 columns (A1-A4), and split matrix B into 4 matrices with 32 rows and 8 columns (B1-B4). Correspondingly, matrix D is also a matrix with 32 rows and 32 columns, and can be split into 16 matrices with 8 rows and 8 columns (D11-D44).

[0053] It can be seen that the matrix A1 is multiplied by the matrix B1 to obtain the matrix D11, the matrix A1 is multiplied by the matrix B2 to obtain the matrix D12, the matrix A1 is multiplied by the matrix B3 to obtain the matrix D13, and the matrix A1 is multiplied by the matrix B4 to obtain the matrix D14.

[0054] Each matrix of 8 rows and 32 columns can be further split into 4 matrices of 8 rows and 8 columns, and each matrix of 32 rows and 8 columns can be split into 4 matrices of 8 rows and 8 columns. Taking matrix A1 and matrix B1 as examples, matrix A1 can be split into matrices A11-A14, and matrix B1 can be split into matrices B11-B14.

[0055] In this case, the operation array consumes 4 computing cycles to complete the multiplication of matrix A1 and matrix B1 to obtain matrix D11. Figure 2b As shown, in the first calculation cycle, the operation array calculates the product of matrix A11 and matrix B11 to obtain matrix D11_1; in the second calculation cycle, the operation array calculates the product of matrix A12 and matrix B12 and the sum of matrix D11_1 to obtain matrix D11_2; in the third calculation cycle, the operation array calculates the product of matrix A13 and matrix B13 and the sum of matrix D11_2 to obtain matrix D11_3; in the fourth calculation cycle, the operation array calculates the product of matrix A14 and matrix B14 and the sum of matrix D11_3 to obtain matrix D11, which is the result of multiplying matrix A1 and matrix B1.

[0056] By analogy, the operation array consumes 64 computing cycles to calculate the matrices D11-D44, that is, it takes a total of 64 computing cycles to complete the multiplication operation of the matrix A and the matrix B.

[0057] Figure 3 An example of calculation completed in multiple calculation cycles by a single calculation unit in a calculation array of the prior art is shown.

[0058] like Figure 3 As shown, in the 1st to 4th calculation cycles, the operation unit can be used to calculate the product of the first row of matrix A1 and the first column of matrix B1. The first row of matrix A1 can be regarded as a vector including 32 elements a1-a32, and the first column of matrix B1 can be regarded as a vector including 32 elements b1-b32. Each element may be 1-bit low-width data, or may be 8-bit or 32-bit high-width data.

[0059] Since each operation unit can complete the multiplication operation of a vector of 1 row and 8 columns and a vector of 8 rows and 1 column in one calculation cycle, the first row of matrix A1 and the first column of matrix B1 can be split into multiple vectors with 8 elements. Then the first row of matrix A1 can be split into 4 vectors including elements a1-a8, elements a9-a16, elements a17-a24, and elements a25-a32, and the first column of matrix B1 can be split into 4 vectors including elements b1-b8, elements b9-b16, elements b17-b24, and elements b25-b32.

[0060] In this case, in the first calculation cycle, the operation unit can be used to calculate the product of the vector including elements a1-a8 and the vector including elements b1-b8 as the calculation result of the operation unit in the first calculation cycle. In the second calculation cycle, the operation unit can be used to calculate the product of the vector including elements a9-a16 and the vector including elements b9-b16, and calculate the sum of the product and the calculation result of the operation unit in the first calculation cycle as the calculation result of the operation unit in the second calculation cycle. In the third calculation cycle, the operation unit can be used to calculate the product of the vector including elements a17-a24 and the vector including elements b17-b24, and calculate the sum of the product and the calculation result of the operation unit in the second calculation cycle as the calculation result of the operation unit in the third calculation cycle. In the 4th calculation cycle, the operation unit can be used to calculate the product of the vector including elements a25-a32 and the vector including elements b25-b32, and calculate the sum of the product and the calculation result of the operation unit in the 3rd calculation cycle as the calculation result of the operation unit in the 4th calculation cycle.

[0061] The operation array can write the calculation results of each operation unit in each calculation cycle into the corresponding position of the matrix D. When entering the next calculation cycle, the controller can read the calculation results written in the previous calculation cycle from the matrix D and transmit them to the operation array. The controller can control the first cache and the second cache to read the data stored in the corresponding position according to the storage position of the vector to be calculated by each operation unit in the current calculation cycle in the matrix A and the matrix B, and output it to the operation array in the form of a vector. The operation array can use the data input from the first cache and the second cache to complete the multiplication operation of the current calculation cycle, and use the data input from the controller to complete the addition operation of the current calculation cycle.

[0062] The disadvantage of the prior art is that the matrix to be used for GEMM is usually sparse data, and the number of calculation cycles consumed by the prior art to complete GEMM is only related to the upper limit of the computing power of the operation array (that is, the number of matrix rows and columns that can complete multiplication operations in one calculation cycle of the operation array) and the number of rows and columns of the matrix to be used for GEMM. The sparsity of the data is not taken into account, and invalid operands (that is, elements with a value of 0) are also involved in the calculation. This results in a large amount of data to be calculated, a high number of GEMM calculation cycles, high computing power consumption of the device, and low computing efficiency and energy efficiency. If you want to improve computing efficiency, you can only increase the computing power of the operation array, which will increase the hardware resource overhead and reduce the energy efficiency of the device. Therefore, computing efficiency and energy efficiency cannot be taken into account at the same time. In addition, the control method of the controller is relatively simple, and the computing flexibility is poor.

[0063] In view of this, the present disclosure proposes a general matrix multiplication operation device, method, and electronic device. The GEMM algorithm implemented by the general matrix multiplication operation device of the present disclosure can reduce the amount of data to be calculated, thereby reducing the computing power consumption of the device and improving the energy efficiency of the device.

[0064] Furthermore, the device can reduce the number of computing cycles, so that the computing efficiency of the device can be improved without increasing the computing power of the computing array, thereby improving the energy efficiency of the device.

[0065] Furthermore, the controller disclosed in the present invention can control the computing array to complete the operations of multiple pairs of vectors in parallel with different numbers of consumed computing cycles. Compared with the controller in the prior art, the control method is more diversified and the computing flexibility is better.

[0066] Figure 4 An exemplary application scenario of the general matrix multiplication operation device according to an embodiment of the present disclosure is shown.

[0067] like Figure 4As shown, the general matrix multiplication operation device of the embodiment of the present disclosure can be set in an electronic device and set on a GPU. When the GPU runs a neural network model, it is necessary to perform a multiplication operation on the first data matrix and the second data matrix. The GPU can use the general matrix multiplication operation device to perform a multiplication operation on the first data matrix and the second data matrix to obtain a third data matrix.

[0068] Those skilled in the art should understand that the general matrix multiplication operation device can also be set on a device with matrix multiplication operation requirements such as an AI accelerator in an electronic device, or set as a separate device in an electronic device. The embodiments of the present disclosure do not limit the specific configuration of the general matrix multiplication operation device in the electronic device.

[0069] Figure 5 A schematic diagram showing the structure of a general matrix multiplication operation device according to an embodiment of the present disclosure.

[0070] like Figure 5 As shown, in a possible implementation, the device is used to perform a general matrix multiplication operation on a first data matrix and a second data matrix to obtain a third data matrix, the first data matrix has m rows and s columns, the second data matrix has s rows and n columns, the third data matrix has m rows and n columns, m, s, and n are positive integers, and the device includes:

[0071] A first storage module, used for storing a first data matrix and a second data matrix;

[0072] A mask generation module, for generating a composite mask corresponding to the i-th row element of the first data matrix and the j-th column element of the second data matrix according to the first data matrix and the second data matrix;

[0073] A vector generation module, used to generate a first vector according to the composite mask and the i-th row elements of the first data matrix, and to generate a second vector according to the composite mask and the j-th column elements of the second data matrix, wherein the number of elements of the first vector is equal to the number of elements of the second vector and is less than s, i≤m, j≤n, and i and j are positive integers;

[0074] A calculation module, used for calculating the product of the first vector and the second vector, where the product is equal to the product of the i-th row element of the first data matrix and the j-th column element of the second data matrix;

[0075] The second storage module is used to store a third data matrix. When the general matrix multiplication operation of the first data matrix and the second data matrix is ​​completed, the element in the i-th row and the j-th column of the third data matrix is ​​the product of the first vector and the second vector.

[0076] For example, the general matrix multiplication operation device is used to perform a general matrix multiplication operation on the first data matrix and the second data matrix to obtain a third data matrix. Therefore, the general matrix multiplication operation device may include a first storage module and a second storage module, the first storage module is used to store the first data matrix and the second data matrix, and the second storage module is used to store the third data matrix. The first data matrix has m rows and s columns, the second data matrix has s rows and n columns, and the third data matrix has m rows and n columns, and m, s, and n are positive integers.

[0077] Optionally, the first storage module may further include a first cache unit and a second cache unit, wherein the first cache unit is used to cache the first data matrix, and the second cache unit is used to cache the second data matrix. The embodiment of the present disclosure does not limit the specific structure of the first storage module.

[0078] The i-th row element of the first data matrix and the j-th column element of the second data matrix may be sparse data. The general matrix multiplication operation device may include a mask generation module for generating a composite mask corresponding to the i-th row element of the first data matrix and the j-th column element of the second data matrix according to the first data matrix and the second data matrix. The composite mask can be used to reduce the sparsity of data. An exemplary generation method of the composite mask is given later.

[0079] The general matrix multiplication operation device may also include a vector generation module for generating a first vector according to a composite mask and the i-th row elements of the first data matrix, and generating a second vector according to the composite mask and the j-th column elements of the second data matrix. The number of elements of the first vector is equal to the number of elements of the second vector and is less than s, so the sparsity of the first vector is lower than the sparsity of the i-th row elements of the first data matrix, and the sparsity of the second vector is lower than the sparsity of the j-th column elements of the second data matrix. Further, the first vector and the second vector may be dense data. Exemplary generation methods of the first vector and the second vector are given later.

[0080] The general matrix multiplication operation device may also include an operation module for calculating the product of the first vector and the second vector. The product is also equal to the product of the i-th row element of the first data matrix and the j-th column element of the second data matrix. Since the sparsity of the first vector is lower than the sparsity of the i-th row element of the first data matrix, and the sparsity of the second vector is lower than the sparsity of the j-th column element of the second data matrix, the amount of data required to calculate the product of the first vector and the second vector must be less than the amount of data required to calculate the product of the i-th row element of the first data matrix and the j-th column element of the second data matrix. Since the amount of data calculated is reduced and the calculation result does not change, the calculation power consumption of the operation module is reduced.

[0081] When the general matrix multiplication operation of the first data matrix and the second data matrix is ​​completed, the element of the i-th row and j-th column in the third data matrix is ​​the product of the first vector and the second vector. In this case, each row element of the first data matrix and each column element of the second data matrix can be processed by reducing the sparsity in the above-mentioned manner to obtain a corresponding pair of vectors, and the obtained vectors can be multiplied, so that accurate calculation results can be obtained with less calculated data. The calculation results can be stored in a specific position in the third data matrix. When all elements of the third data matrix are calculated, the general matrix multiplication operation device completes the general matrix multiplication operation of the first data matrix and the second data matrix.

[0082] According to the general matrix multiplication operation device of the embodiment of the present disclosure, the first storage module stores the first data matrix and the second data matrix, the mask generation module generates a composite mask corresponding to the i-th row elements of the first data matrix and the j-th column elements of the second data matrix according to the first data matrix and the second data matrix, the vector generation module generates a first vector according to the composite mask and the i-th row elements of the first data matrix, and generates a second vector according to the composite mask and the j-th column elements of the second data matrix, the number of elements of the first vector is equal to the number of elements of the second vector and is less than s, i≤m, j≤n, i and j are positive integers, so that the sparsity of the first vector is lower than the sparsity of the i-th row elements of the first data matrix, and the sparsity of the second vector is lower than the sparsity of the j-th column elements of the second data matrix; the operation module calculates the product of the first vector and the second vector, and the product is equal to the product of the i-th row elements of the first data matrix and the j-th column elements of the second data matrix, so that accurate calculation results can be obtained with less calculated data. The second storage module stores a third data matrix, and when the general matrix multiplication operation of the first data matrix and the second data matrix is ​​completed, the element of the i-th row and the j-th column in the third data matrix is ​​the product of the first vector and the second vector, so that the third data matrix can be used as the operation result of the general matrix multiplication operation of the first data matrix and the second data matrix. The GEMM algorithm implemented by the general matrix multiplication operation device of the present disclosure takes into account the sparsity of the elements of each matrix multiplied, which can reduce the amount of data to be calculated, thereby reducing the computing power consumption of the device and improving the energy efficiency of the device.

[0083] The following introduces an exemplary structure of the mask generation module according to an embodiment of the present disclosure.

[0084] In a possible implementation, the mask generation module includes:

[0085] A mask matrix generating unit, configured to generate a first mask matrix according to the first data matrix, and to generate a second mask matrix according to the second data matrix;

[0086] A first mask storage unit, used for storing a first mask matrix;

[0087] A second mask storage unit, used for storing a second mask matrix;

[0088] A composite mask generating unit, used to generate a composite mask according to the i-th row elements of the first mask matrix and the j-th column elements of the second mask matrix;

[0089] Wherein, the first mask matrix has m rows and s columns, the second mask matrix has s rows and n columns, the element in the i-th row and k-th column of the first mask matrix indicates whether the element in the i-th row and k-th column of the first data matrix is ​​0, the element in the k-th row and j-th column of the second mask matrix indicates whether the element in the k-th row and j-th column of the second data matrix is ​​0, k≤s, and k is a positive integer;

[0090] The composite mask includes s elements, and the kth element of the composite mask indicates whether the product of the element in the i-th row and k-th column of the first data matrix and the element in the k-th row and j-th column of the second data matrix is ​​0.

[0091] Figure 6 A schematic diagram showing the structure of a mask generation module according to an embodiment of the present disclosure is shown. Figure 6 As shown, the mask generation module may include a mask matrix generation unit, a first mask storage unit, a second mask storage unit, and a composite mask generation unit.

[0092] The mask matrix generating unit may generate a first mask matrix according to the first data matrix, wherein the number of rows of the first mask matrix is ​​m and the number of columns is s, that is, the number of rows and columns of the first mask matrix is ​​the same as the number of rows and columns of the first data matrix. The element of the i-th row and k-th column in the first mask matrix may indicate whether the element of the i-th row and k-th column in the first data matrix is ​​0.

[0093] Similarly, the mask matrix generating unit can generate a second mask matrix according to the second data matrix, the second mask matrix having s rows and n columns, that is, the number of rows and columns of the second mask matrix is ​​the same as the number of rows and columns of the second data matrix. The element in the kth row and jth column of the second mask matrix indicates whether the element in the kth row and jth column of the second data matrix is ​​0.

[0094] The mask matrix generation unit may store the generated first mask matrix into the first mask storage unit, and store the generated second mask matrix into the second mask storage unit. The first mask storage unit and the second mask storage unit may be implemented by using a cache.

[0095] The composite mask generation unit can read the i-th row element of the first mask matrix from the first mask storage unit, and read the j-th column element of the second mask matrix from the second mask storage unit. The composite mask generation unit can generate a composite mask corresponding to the i-th row element of the first data matrix and the j-th column element of the second data matrix according to the i-th row element of the first mask matrix and the j-th column element of the second mask matrix, the composite mask including s elements, the k-th element of the composite mask indicating whether the product of the i-th row and k-th column element of the first data matrix and the k-th row and j-th column element of the second data matrix is ​​0.

[0096] When the product of the element in the i-th row and k-th column of the first data matrix and the element in the k-th row and j-th column of the second data matrix is ​​0, it can be considered that the element in the i-th row and k-th column of the first data matrix and the element in the k-th row and j-th column of the second data matrix are invalid operands for calculating the product of the element in the i-th row of the first data matrix and the element in the j-th column of the second data matrix. In this case, when the vector generation module generates the first vector and the second vector, the invalid operands can be removed according to the composite mask, so that the sparsity of the first vector is lower than the element in the i-th row of the first data matrix, and the sparsity of the second vector is lower than the element in the j-th column of the second data matrix.

[0097] An exemplary method for generating a composite mask is described below.

[0098] Figure 7 Examples of elements in a first data matrix, a second data matrix, a first mask matrix, a second mask matrix, and a composite mask according to an embodiment of the present disclosure are shown.

[0099] In a possible implementation, when the element in the i-th row and the k-th column in the first mask matrix is ​​0, it indicates that the element in the i-th row and the k-th column in the first data matrix is ​​also 0;

[0100] When the element in the i-th row and the k-th column in the first mask matrix is ​​1, it indicates that the element in the i-th row and the k-th column in the first data matrix is ​​not 0;

[0101] When the element in the k-th row and j-th column in the second mask matrix is ​​0, it indicates that the element in the k-th row and j-th column in the second data matrix is ​​also 0;

[0102] When the element in the k-th row and j-th column in the second mask matrix is ​​1, it indicates that the element in the k-th row and j-th column in the second data matrix is ​​not 0.

[0103] For example, each element in the first mask matrix and the second mask matrix only needs to occupy 1 bit. Therefore, when the element in the i-th row and k-th column of the first mask matrix is ​​0, it indicates that the element in the i-th row and k-th column of the first data matrix is ​​also 0, and when the element in the i-th row and k-th column of the first mask matrix is ​​1, it indicates that the element in the i-th row and k-th column of the first data matrix is ​​not 0.

[0104] like Figure 7 As shown, assuming that the first data matrix is ​​the matrix A mentioned above, the second data matrix is ​​the matrix B mentioned above, and the third data matrix is ​​the matrix D mentioned above, then m=n=s=32.

[0105] Assume i=1, in the first row of the first data matrix, a1, a2, a5, a6, a8, a9, a10, a13, a14, a17, a19, a23, a24, a27, a29, a30 are not 0, and the rest are 0. Then in the first row of the first mask matrix, the 1st, 2nd, 5th, 6th, 8th, 9th, 10th, 13th, 14th, 17th, 19th, 23rd, 24th, 27th, 29th, 30th elements are 1, and the rest are 0. The elements in the first row of the first mask matrix are 11001101 1100 1100 1010 0011 0010 1100.

[0106] Similarly, when the element in the kth row and jth column in the second mask matrix is ​​0, it indicates that the element in the kth row and jth column in the second data matrix is ​​also 0, and when the element in the kth row and jth column in the second mask matrix is ​​1, it indicates that the element in the kth row and jth column in the second data matrix is ​​not 0.

[0107] Assume j=1, in the first column of the second data matrix, b1, b2, b3, b4, b5, b6, b8, b10, b13, b15, b16, b17, b19, b21, b23, b24, b27, b28, b30 are not 0, and the rest are 0. Then in the first column of the second mask matrix, the 1st, 2nd, 3rd, 4th, 5th, 6th, 8th, 10th, 13th, 15th, 16th, 17th, 19th, 21st, 23rd, 24th, 27th, 28th, 30th elements are 1, and the rest are 0, and the elements in the first column of the second mask matrix are 1111 1101 0100 1011 1010 1011 0011 0100.

[0108] In a possible implementation, the composite mask generating unit is specifically configured to:

[0109] Performing an AND operation on the i-th row element of the first mask matrix and the j-th column element of the second mask matrix to obtain a composite mask;

[0110] When the kth element of the composite mask is 0, it indicates that the product of the element in the i-th row and k-th column of the first data matrix and the element in the k-th row and j-th column of the second data matrix is ​​0. When the kth element of the composite mask is 1, it indicates that the product of the element in the i-th row and k-th column of the first data matrix and the element in the k-th row and j-th column of the second data matrix is ​​not 0.

[0111] For example, when any one of the two elements is 0, the product of the two elements is equal to 0, and the result of the AND operation of the two elements is also equal to 0; when neither of the two elements is 0, the product of the two elements is also not 0, and the result of the AND operation of the two elements is equal to 1. Therefore, the result of the AND operation of the two elements can indicate whether the product of the two elements is 0.

[0112] In this case, an AND operation can be performed on the i-th row elements of the first mask matrix and the j-th column elements of the second mask matrix to obtain a composite mask corresponding to the i-th row elements of the first data matrix and the j-th column elements of the second data matrix. When the k-th element of the composite mask is 0, it indicates that the product of the i-th row and k-th column elements in the first mask matrix and the k-th row and j-th column elements in the second mask matrix is ​​0, and further, it also indicates that the product of the i-th row and k-th column elements in the first data matrix and the k-th row and j-th column elements in the second data matrix is ​​0. When the k-th element of the composite mask is 1, it indicates that the product of the i-th row and k-th column elements in the first mask matrix and the k-th row and j-th column elements in the second mask matrix is ​​not 0, and further, it also indicates that the product of the i-th row and k-th column elements in the first data matrix and the k-th row and j-th column elements in the second data matrix is ​​not 0.

[0113] like Figure 7 As shown, the first row elements of the first mask matrix are: 1100 1101 1100 1100 1010 0011 0010 1100;

[0115] The first column elements of the second mask matrix are: 1111 1101 0100 1011 1010 1011 0011 0100;

[0117] The result of the AND operation of the first row element of the first mask matrix and the first column element of the second mask matrix, that is, the composite mask corresponding to the first row element of the first mask matrix and the first column element of the second mask matrix, can be:

[0118] 1100 1101 0100 1000 1010 0011 0010 0100.

[0119] It can be seen that when the kth element of the composite mask is 0, at least one of the elements in the 1st row and kth column of the first mask matrix and the elements in the kth row and 1st column of the second mask matrix is ​​also 0, that is, at least one of the elements in the 1st row and kth column of the first data matrix and the elements in the kth row and 1st column of the second data matrix is ​​0, and the product of the elements in the 1st row and kth column of the first data matrix and the elements in the kth row and 1st column of the second data matrix is ​​0.

[0120] When the kth element of the composite mask is 1, the element in the 1st row and kth column of the first mask matrix and the element in the kth row and 1st column of the second mask matrix are both not 0, that is, the element in the 1st row and kth column of the first data matrix and the element in the kth row and 1st column of the second data matrix are also not 0. The product of the element in the 1st row and kth column of the first data matrix and the element in the kth row and 1st column of the second data matrix is ​​also not 0.

[0121] In this way, the composite mask can be obtained accurately.

[0122] An exemplary manner in which the vector generation module generates the first vector and the second vector is introduced below.

[0123] In a possible implementation, the vector generation module is specifically used to:

[0124] Performing an AND operation on the composite mask and the i-th row elements of the first data matrix, removing the elements with a value of 0 in the operation result and densely arranging the remaining elements to obtain a first vector;

[0125] A logical AND operation is performed on the composite mask and the elements of the j-th column of the second data matrix, and the elements with a value of 0 in the operation result are removed and the remaining elements are densely arranged to obtain a second vector.

[0126] For example, the vector generation module may receive a composite mask from the composite mask generation unit. When the composite mask corresponds to the i-th row element of the first data matrix and the j-th column element of the second data matrix, the vector generation module may perform an AND operation on the composite mask and the i-th row element of the first data matrix, wherein the k-th element of the composite mask is ANDed with the k-th element in the i-th row element of the first data matrix. Since each element of the composite mask is either 1 or 0, the result of the AND operation of the composite mask and the i-th row element of the first data matrix will consist of a portion of the i-th row element of the first data matrix and 0.

[0127] The AND operation result data being 0 indicates that the element obtained by calculating the AND operation result is an invalid operand for calculating the product of the i-th row element of the first data matrix and the j-th column element of the second data matrix. Therefore, the first vector can be obtained by removing the elements with a value of 0 in the AND operation result and densely arranging the remaining elements.

[0128] Similarly, a logical AND operation is performed on the composite mask and the elements of the j-th column of the second data matrix, and the elements with a value of 0 in the AND operation result are removed and the remaining elements are densely arranged to obtain the second vector.

[0129] Optionally, the vector generation module may include a first vector generation unit and a second vector generation unit. The first vector generation unit is used to generate the first vector, and the second generation unit is used to generate the second vector. The present disclosure does not limit the specific structure of the vector generation module.

[0130] Figure 8 Examples of elements in a first data matrix, a second data matrix, a composite mask, a first vector, and a second vector according to an embodiment of the present disclosure are shown.

[0131] like Figure 8 As shown, the composite mask corresponding to the first row element of the first mask matrix and the first column element of the second mask matrix can be: 1100 1101 0100 1000 1010 0011 0010 0100. The first vector obtained by the AND operation of the composite mask and the first row element of the first data matrix can be [a1, a2, a5, a6, a8, a10, a13, a17, a19, a23, a24, a27, a30]. The second vector obtained by the AND operation of the composite mask and the first column element of the second data matrix can be [b1, b2, b5, b6, b8, b10, b13, b17, b19, b23, b24, b27, b30].

[0132] The following is an exemplary method for reducing the number of calculation cycles of the general matrix multiplication operation device according to an embodiment of the present disclosure.

[0133] In a possible implementation, the device further includes a controller, and the operation module completes multiplication operations of multiple pairs of vectors with a number of elements w in one calculation cycle, where w is a positive integer less than s;

[0134] The controller is used for splitting the first vector and the second vector according to the value of w, the first vector is split into z first sub-vectors, the second vector is split into z second sub-vectors, the t-th first sub-vector and the t-th second sub-vector include the same number of elements, t≤z, t and z are positive integers;

[0135] The number of elements of the 1st to z-1th first sub-vectors is equal to w, and the number of elements of the zth first sub-vector is less than or equal to w;

[0136] The operation module is specifically used to calculate the product of the first vector and the second vector through z calculation cycles, and in the tth calculation cycle, write the calculation result of the tth calculation cycle into the i-th row and j-th column of the third data matrix, the calculation result of the tth calculation cycle is equal to the sum of the product of the tth first sub-vector and the tth second sub-vector and the calculation result of the t-1th calculation cycle, and the calculation result of the zth calculation cycle is the product of the first vector and the second vector.

[0137] Fig. 9 A schematic diagram showing the structure of a general matrix multiplication operation device according to an embodiment of the present disclosure.

[0138] For example, if Fig. 9As shown, the general matrix multiplication operation device may further include a controller for controlling the cyclic accumulation of multiple operation cycles of the GEMM algorithm.

[0139] The controller can be used to control the first storage module to output the first data matrix and the second data matrix to the mask generation module, and the mask generation module generates a composite mask corresponding to each row of the first data matrix and each column of the second data matrix.

[0140] The controller can also control the mask generation module to input the composite mask into the vector generation module, and control the first storage module to output a row of elements in the first data matrix corresponding to the composite mask and a column of elements in the second data matrix corresponding to the composite mask to the vector generation module, and the vector generation module will obtain a pair of first vectors and second vectors.

[0141] The controller may also split the first vector and the second vector according to the value of w, the first vector is split into z first sub-vectors, the second vector is split into z second sub-vectors, the t-th first sub-vector and the t-th second sub-vector include the same number of elements, t≤z, t and z are positive integers. The number of elements of the 1st to z-1th first sub-vectors is equal to w, and the number of elements of the z-th first sub-vector is less than or equal to w.

[0142] Assume that the operation module completes the multiplication operation of multiple pairs of vectors with w elements in one calculation cycle, where w is a positive integer less than s. In this case, the operation module can complete the multiplication operation of the first vector and the second vector in z calculation cycles. Wherein, in the t-th calculation cycle, the calculation result of the t-th calculation cycle is written into the i-th row and j-th column of the third data matrix, the calculation result of the t-th calculation cycle is equal to the sum of the product of the t-th first sub-vector and the t-th second sub-vector and the calculation result of the t-1-th calculation cycle, and the calculation result of the z-th calculation cycle is the product of the first vector and the second vector.

[0143] In one possible implementation, in the tth calculation cycle, the controller is further configured to:

[0144] The control vector generation module outputs the t-th first sub-vector and the t-th second sub-vector to the operation module;

[0145] When t is greater than 1, the second storage module is also controlled to output the calculation result of the t-1th calculation cycle to the operation module.

[0146] For example, which data is input into the operation module in each calculation cycle is also controlled by the controller. In the tth calculation cycle, the operation module calculates the sum of the product of the tth first sub-vector and the tth second sub-vector and the calculation result of the t-1th calculation cycle. Therefore, the controller can be used to control the vector generation module to output the tth first sub-vector and the tth second sub-vector to the operation module. Since there is no calculation result of the 0th calculation cycle when t=1, there is no need to input the data stored in the second storage module into the operation module when t=1. When t is greater than 1, there is a calculation result of the t-1th calculation cycle and the result is stored in the second storage module. Therefore, the controller can also control the second storage module to output the calculation result of the t-1th calculation cycle to the operation module when t is greater than 1.

[0147] The following introduces an exemplary structure of the operation module of the embodiment of the present disclosure.

[0148] In a possible implementation, the operation module includes a plurality of operation units, each of which completes a multiplication operation of a pair of vectors with w elements in one calculation cycle.

[0149] When any one of the operation units is used to calculate the product of the first vector and the second vector, the operation unit is specifically used to:

[0150] In the first calculation cycle, the product of the first first sub-vector and the first second sub-vector is calculated as the calculation result of the first calculation cycle, and the calculation result of the first calculation cycle is written into the i-th row and j-th column of the third data matrix;

[0151] When t is greater than 1, in the tth calculation cycle, the product of the tth first sub-vector and the tth second sub-vector is calculated, and the sum of the product and the calculation result of the t-1th calculation cycle is used as the calculation result of the tth calculation cycle, and the calculation result of the tth calculation cycle is written into the i-th row and j-th column of the third data matrix.

[0152] For example, the operation module may include multiple operation units, each of which completes a multiplication operation of a pair of vectors with w elements in one calculation cycle. Therefore, the operation module can simultaneously complete multiplication operations of multiple pairs of vectors with w elements in one calculation cycle.

[0153] Fig.10 A schematic diagram showing the structure of a computing unit according to an embodiment of the present disclosure.

[0154] like Fig.10As shown, assuming w=8, the operation unit may include 8 multipliers and 8 adders. Each multiplier is used to calculate the product of the elements at the same position of two vectors, and the 8 multipliers respectively calculate the product of the first element of the two vectors, ..., the product of the eighth element of the two vectors, and a total of 8 products c1-c8 are obtained. Adder 1 is used to calculate the sum c12 of product c1 and product c2, adder 2 is used to calculate the sum c34 of product c3 and product c4, adder 3 is used to calculate the sum c56 of product c5 and product c6, and adder 4 is used to calculate the sum c78 of product c7 and product c8. Adder 5 is used to calculate the sum c1234 of c12 and c34, and adder 6 is used to calculate the sum c5678 of c56 and c78. Adder 7 is used to calculate the sum c0 of c1234 and c5678. c0 is the result of multiplication of a pair of vectors with 8 elements.

[0155] Adder 8 can be used to calculate the sum of c0 and the calculation result of the previous calculation cycle. In each calculation cycle, 8 multipliers and adders 1 to 7 are used. Adder 8 is used when the number of calculation cycles is greater than 1.

[0156] Fig.11 An example is shown in which a calculation unit according to an embodiment of the present disclosure calculates the product of a first vector and a second vector through multiple calculation cycles.

[0157] like Fig.11 As shown, the number of elements included in the first vector and the second vector is equal to 13. Assuming w=8, the first vector can be split into 2 (z=2) first sub-vectors, the number of elements of the first first sub-vector is equal to 8, and the number of elements of the second first sub-vector is equal to 5. The second vector can be split into 2 second sub-vectors, the number of elements of the first second sub-vector is equal to 8, and the number of elements of the second second sub-vector is equal to 5.

[0158] The operation unit can calculate the product of the first vector and the second vector through two calculation cycles. In the first calculation cycle, the operation unit calculates the product of the first first sub-vector and the first second sub-vector as the calculation result of the first calculation cycle. In the second calculation cycle, the operation unit calculates the product of the second first sub-vector and the second second sub-vector, and calculates the sum of the product and the calculation result of the first calculation cycle as the calculation result of the second calculation cycle, which is also the product of the first vector and the second vector.

[0159] In this case, the product that needs 4 calculation cycles to be calculated according to the prior art solution can be calculated in only 2 calculation cycles in the present disclosure, which can reduce the number of calculation cycles of GEMM.

[0160] The present disclosure also proposes a general matrix multiplication method, Fig.12A schematic diagram showing the process of a general matrix multiplication method according to an embodiment of the present disclosure.

[0161] like Fig.12 As shown, in a possible implementation, the method is applied to a general matrix multiplication operation device, the device is used to perform a general matrix multiplication operation on a first data matrix and a second data matrix to obtain a third data matrix, the first data matrix has m rows and s columns, the second data matrix has s rows and n columns, the third data matrix has m rows and n columns, m, s, and n are positive integers, the device includes a first storage module, a mask generation module, a vector generation module, an operation module, and a second storage module, and the method includes:

[0162] Step S120, using a first storage module to store the first data matrix and the second data matrix;

[0163] Step S121, using a mask generation module to generate a composite mask corresponding to the i-th row element of the first data matrix and the j-th column element of the second data matrix according to the first data matrix and the second data matrix;

[0164] Step S122, using a vector generation module to generate a first vector according to the composite mask and the i-th row elements of the first data matrix, and to generate a second vector according to the composite mask and the j-th column elements of the second data matrix, wherein the number of elements of the first vector is equal to the number of elements of the second vector and is less than s, i≤m, j≤n, and i and j are positive integers;

[0165] Step S123, using a calculation module to calculate the product of the first vector and the second vector, the product being equal to the product of the i-th row element of the first data matrix and the j-th column element of the second data matrix;

[0166] Step S124, using the second storage module to store the third data matrix, when the general matrix multiplication operation of the first data matrix and the second data matrix is ​​completed, the element in the i-th row and j-th column of the third data matrix is ​​the product of the first vector and the second vector.

[0167] In a possible implementation, the mask generation module includes a mask matrix generation unit, a first mask storage unit, a second mask storage unit, and a composite mask generation unit, wherein the generation of a composite mask corresponding to the i-th row element of the first data matrix and the j-th column element of the second data matrix according to the first data matrix and the second data matrix includes: using the mask matrix generation unit to generate a first mask matrix according to the first data matrix, and generating a second mask matrix according to the second data matrix; using the first mask storage unit to store the first mask matrix; using the second mask storage unit to store the second mask matrix; using the composite mask generation unit to generate a composite mask corresponding to the i-th row element of the first data matrix and the j-th column element of the second data matrix; The i-th row elements of the matrix and the j-th column elements of the second mask matrix generate the composite mask; wherein the first mask matrix has m rows and s columns, the second mask matrix has s rows and n columns, the elements in the i-th row and k-th column of the first mask matrix indicate whether the elements in the i-th row and k-th column of the first data matrix are 0, and the elements in the k-th row and j-th column of the second mask matrix indicate whether the elements in the k-th row and j-th column of the second data matrix are 0, k≤s, k is a positive integer; the composite mask includes s elements, and the k-th element of the composite mask indicates whether the product of the elements in the i-th row and k-th column of the first data matrix and the elements in the k-th row and j-th column of the second data matrix is ​​0.

[0168] In a possible implementation, when the element in the i-th row and k-th column of the first mask matrix is ​​0, it indicates that the element in the i-th row and k-th column of the first data matrix is ​​also 0; when the element in the i-th row and k-th column of the first mask matrix is ​​1, it indicates that the element in the i-th row and k-th column of the first data matrix is ​​not 0; when the element in the k-th row and j-th column of the second mask matrix is ​​0, it indicates that the element in the k-th row and j-th column of the second data matrix is ​​also 0; when the element in the k-th row and j-th column of the second mask matrix is ​​1, it indicates that the element in the k-th row and j-th column of the second data matrix is ​​not 0.

[0169] In a possible implementation, the generating the composite mask according to the i-th row elements of the first mask matrix and the j-th column elements of the second mask matrix includes: performing an AND operation on the i-th row elements of the first mask matrix and the j-th column elements of the second mask matrix to obtain the composite mask; when the k-th element of the composite mask is 0, it indicates that the product of the elements in the i-th row and k-th column of the first data matrix and the elements in the k-th row and j-th column of the second data matrix is ​​0; when the k-th element of the composite mask is 1, it indicates that the product of the elements in the i-th row and k-th column of the first data matrix and the elements in the k-th row and j-th column of the second data matrix is ​​not 0.

[0170] In a possible implementation, generating a first vector according to the composite mask and the i-th row elements of the first data matrix, and generating a second vector according to the composite mask and the j-th column elements of the second data matrix, includes: performing an AND operation on the composite mask and the i-th row elements of the first data matrix, removing elements with a value of 0 in the operation result and densely arranging the remaining elements to obtain the first vector; performing a logical AND operation on the composite mask and the j-th column elements of the second data matrix, removing elements with a value of 0 in the operation result and densely arranging the remaining elements to obtain the second vector.

[0171] In a possible implementation, the device further includes a controller, the operation module completes a plurality of multiplication operations of vectors having a number of elements of w in one calculation cycle, where w is a positive integer less than s, and the method further includes: using the controller to split the first vector and the second vector according to the value of w, the first vector is split into z first sub-vectors, the second vector is split into z second sub-vectors, the t-th first sub-vector and the t-th second sub-vector include the same number of elements, t≤z, t and z are positive integers; the number of elements of the 1st to z-1th first sub-vectors is equal to w, the number of elements of the zth first sub-vector is less than or equal to w; the calculating the product of the first vector and the second vector includes: calculating the product of the first vector and the second vector through z calculation cycles, and in the tth calculation cycle, writing the calculation result of the tth calculation cycle into the i-th row and j-th column of the third data matrix, the calculation result of the tth calculation cycle is equal to the sum of the product of the tth first sub-vector and the tth second sub-vector and the calculation result of the t-1th calculation cycle, and the calculation result of the zth calculation cycle is the product of the first vector and the second vector.

[0172] In a possible implementation, the method further includes: in the tth calculation cycle, using the controller to control the vector generation module to output the tth first sub-vector and the tth second sub-vector to the operation module; when t is greater than 1, using the controller to control the second storage module to output the calculation result of the t-1th calculation cycle to the operation module.

[0173] In a possible implementation, the operation module includes multiple operation units, each operation unit completes a multiplication operation of a pair of vectors with w elements in one calculation cycle, and the product of the first vector and the second vector is calculated through z calculation cycles, including: when any operation unit is used to calculate the product of the first vector and the second vector, in the first calculation cycle, the operation unit is used to calculate the product of the first first sub-vector and the first second sub-vector, as the calculation result of the first calculation cycle, and the calculation result of the first calculation cycle is written into the i-th row and j-th column of the third data matrix; when t is greater than 1, in the t-th calculation cycle, the operation unit is used to calculate the product of the t-th first sub-vector and the t-th second sub-vector, the sum of the product and the calculation result of the t-1-th calculation cycle is used as the calculation result of the t-th calculation cycle, and the calculation result of the t-th calculation cycle is written into the i-th row and j-th column of the third data matrix.

[0174] The present disclosure also provides an electronic device, including the above-mentioned general matrix multiplication operation device. The schematic diagram of the structure of the electronic device can be seen in Figure 4 .

[0175] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0176] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A general matrix multiplication operation device, characterized in that: The device is used to perform a general matrix multiplication operation on a first data matrix and a second data matrix to obtain a third data matrix, wherein the first data matrix has m rows and s columns, the second data matrix has s rows and n columns, and the third data matrix has m rows and n columns, where m, s, and n are positive integers, and the device comprises: A first storage module, used for storing the first data matrix and the second data matrix; a mask generating module, configured to generate a composite mask corresponding to the i-th row element of the first data matrix and the j-th column element of the second data matrix according to the first data matrix and the second data matrix; a vector generation module, configured to generate a first vector according to the composite mask and the i-th row elements of the first data matrix, and to generate a second vector according to the composite mask and the j-th column elements of the second data matrix, wherein the number of elements of the first vector is equal to the number of elements of the second vector and is less than s, i≤m, j≤n, and i and j are positive integers; A calculation module, used for calculating the product of the first vector and the second vector, where the product is equal to the product of the i-th row element of the first data matrix and the j-th column element of the second data matrix; The second storage module is used to store the third data matrix. When the general matrix multiplication operation of the first data matrix and the second data matrix is ​​completed, the element in the i-th row and the j-th column of the third data matrix is ​​the product of the first vector and the second vector.

2. The device according to claim 1, characterized in that The mask generation module comprises: a mask matrix generating unit, configured to generate a first mask matrix according to the first data matrix, and to generate a second mask matrix according to the second data matrix; A first mask storage unit, used to store the first mask matrix; A second mask storage unit, used to store the second mask matrix; a composite mask generating unit, configured to generate the composite mask according to the i-th row elements of the first mask matrix and the j-th column elements of the second mask matrix; The first mask matrix has m rows and s columns, the second mask matrix has s rows and n columns, the element in the i-th row and k-th column of the first mask matrix indicates whether the element in the i-th row and k-th column of the first data matrix is ​​0, the element in the k-th row and j-th column of the second mask matrix indicates whether the element in the k-th row and j-th column of the second data matrix is ​​0, k≤s, and k is a positive integer; The composite mask includes s elements, and the kth element of the composite mask indicates whether the product of the element in the i-th row and k-th column of the first data matrix and the element in the k-th row and j-th column of the second data matrix is ​​0.

3. The device according to claim 2, characterized in that When the element in the i-th row and the k-th column in the first mask matrix is ​​0, it indicates that the element in the i-th row and the k-th column in the first data matrix is ​​also 0; When the element in the i-th row and the k-th column in the first mask matrix is ​​1, it indicates that the element in the i-th row and the k-th column in the first data matrix is ​​not 0; When the element in the k-th row and the j-th column in the second mask matrix is ​​0, it indicates that the element in the k-th row and the j-th column in the second data matrix is ​​also 0; When the element in the k-th row and j-th column in the second mask matrix is ​​1, it indicates that the element in the k-th row and j-th column in the second data matrix is ​​not 0.

4. The device according to claim 2, characterized in that The composite mask generating unit is specifically used for: Performing an AND operation on the i-th row elements of the first mask matrix and the j-th column elements of the second mask matrix to obtain the composite mask; When the kth element of the composite mask is 0, it indicates that the product of the element in the i-th row and k-th column of the first data matrix and the element in the k-th row and j-th column of the second data matrix is ​​0. When the kth element of the composite mask is 1, it indicates that the product of the element in the i-th row and k-th column of the first data matrix and the element in the k-th row and j-th column of the second data matrix is ​​not 0.

5. The device according to claim 4, characterized in that The vector generation module is specifically used for: Performing an AND operation on the composite mask and the elements of the i-th row of the first data matrix, removing the elements with a value of 0 in the operation result, and densely arranging the remaining elements to obtain the first vector; A logical AND operation is performed on the composite mask and the elements of the j-th column of the second data matrix, and the elements with a value of 0 in the operation result are removed and the remaining elements are densely arranged to obtain the second vector.

6. The device according to claim 1, characterized in that The device also includes a controller, and the operation module completes multiplication operations of multiple pairs of vectors with a number of elements of w in one calculation cycle, where w is a positive integer less than s; The controller is used to split the first vector and the second vector according to the value of w, the first vector is split into z first sub-vectors, the second vector is split into z second sub-vectors, the t-th first sub-vector and the t-th second sub-vector include the same number of elements, t≤z, t and z are positive integers; The number of elements of the 1st to z-1th first sub-vectors is equal to w, and the number of elements of the zth first sub-vector is less than or equal to w; The operation module is specifically used to calculate the product of the first vector and the second vector through z calculation cycles, and in the tth calculation cycle, write the calculation result of the tth calculation cycle into the i-th row and j-th column of the third data matrix, the calculation result of the tth calculation cycle is equal to the sum of the product of the tth first sub-vector and the tth second sub-vector and the calculation result of the t-1th calculation cycle, and the calculation result of the zth calculation cycle is the product of the first vector and the second vector.

7. The device according to claim 6, characterized in that In the tth calculation cycle, the controller is also used to, Controlling the vector generation module to output the t-th first sub-vector and the t-th second sub-vector to the operation module; When t is greater than 1, the second storage module is also controlled to output the calculation result of the t-1th calculation cycle to the operation module.

8. The device according to claim 6, characterized in that The operation module includes a plurality of operation units, each of which completes a multiplication operation of a pair of vectors with w elements in one calculation cycle. When any one of the operation units is used to calculate the product of the first vector and the second vector, the operation unit is specifically used to: In the first calculation cycle, the product of the first first sub-vector and the first second sub-vector is calculated as the calculation result of the first calculation cycle, and the calculation result of the first calculation cycle is written into the i-th row and j-th column of the third data matrix; When t is greater than 1, in the tth calculation cycle, the product of the tth first sub-vector and the tth second sub-vector is calculated, and the sum of the product and the calculation result of the t-1th calculation cycle is used as the calculation result of the tth calculation cycle, and the calculation result of the tth calculation cycle is written into the i-th row and j-th column of the third data matrix.

9. A general matrix multiplication method, characterized in that: The method is applied to a general matrix multiplication operation device, the device is used to perform a general matrix multiplication operation on a first data matrix and a second data matrix to obtain a third data matrix, the first data matrix has m rows and s columns, the second data matrix has s rows and n columns, the third data matrix has m rows and n columns, m, s, and n are positive integers, the device comprises a first storage module, a mask generation module, a vector generation module, an operation module, and a second storage module, and the method comprises: Using the first storage module to store the first data matrix and the second data matrix; Using the mask generation module to generate a composite mask corresponding to the i-th row element of the first data matrix and the j-th column element of the second data matrix according to the first data matrix and the second data matrix; Generate a first vector using the vector generation module according to the composite mask and the i-th row elements of the first data matrix, and generate a second vector according to the composite mask and the j-th column elements of the second data matrix, wherein the number of elements of the first vector is equal to the number of elements of the second vector and is less than s, i≤m, j≤n, and i and j are positive integers; Using the operation module to calculate the product of the first vector and the second vector, the product being equal to the product of the i-th row element of the first data matrix and the j-th column element of the second data matrix; The second storage module is used to store the third data matrix. When the general matrix multiplication operation of the first data matrix and the second data matrix is ​​completed, the element in the i-th row and j-th column of the third data matrix is ​​the product of the first vector and the second vector.

10. An electronic device, characterized in that: A general matrix multiplication operation device comprising any one of claims 1-8.

Images