Matrix multiplication circuit, method and related products

Abstract

The present disclosure discloses a matrix multiplication circuit, a method for performing sparse matrix multiplication using the matrix multiplication circuit, and related products. The matrix multiplication circuit can be implemented as a computing device included in a combined processing device, and the combined processing device can also include an interface device and other processing devices. The computing device interacts with the other processing devices to jointly complete the computing operations specified by the user. The combined processing device can also include a storage device, which is respectively connected to the computing device and the other processing devices and is used to store data from the computing device and the other processing devices. The solution disclosed in the present disclosure provides a circuit that supports sparse matrix multiplication, which can simplify processing and improve the processing efficiency of the machine.

Description

Technical Field

[0001] The present disclosure generally relates to the field of data processing, and more particularly to a matrix multiplication circuit, a method for performing sparse matrix multiplication using the matrix multiplication circuit, a chip, and a board. Background Art

[0002] Sparse matrices play a significant role in data processing. For example, deep learning algorithms, which have rapidly developed in recent years, are computationally and storage-intensive tools. As information processing tasks become increasingly complex, the demands for real-time performance and accuracy are constantly increasing. Neural networks are often designed to be deeper, increasing the computational load and storage space requirements. This makes existing deep learning-based artificial intelligence technologies difficult to directly apply to mobile phones, satellites, or embedded devices with limited hardware resources.

[0003] Therefore, compressing, accelerating, and optimizing deep neural network models has become extremely important. Sparsification is one method for lightweighting models. Network parameter sparsification involves reducing redundant components in larger networks through appropriate methods, thereby reducing the network's computational load and storage requirements. This network parameter sparsification results in a sparse matrix.

[0004] The properties of sparse matrices dictate that they can leverage their data structure characteristics to significantly reduce computational and storage complexity. Therefore, research and hardware implementation of sparse matrix storage and computation methods can significantly improve processing performance for these data structures. However, existing hardware cannot fully exploit the characteristics of sparse matrices and effectively support sparse matrix-related operations. Summary of the Invention

[0005] In order to at least partially solve one or more technical problems mentioned in the background technology, the solution disclosed herein provides a matrix multiplication circuit, a method for performing sparse matrix multiplication using the matrix multiplication circuit, a chip, and a board.

[0006] In a first aspect, the present disclosure discloses a matrix multiplication circuit, wherein at least one of the left multiplication matrix and the right multiplication matrix is ​​a sparse matrix, and the circuit includes: a segmentation circuit, configured to divide the i-th row vector of the left multiplication matrix and the j-th column vector of the right multiplication matrix into a plurality of paired vector segments, where i and j are natural numbers; a control circuit, configured to discard the corresponding vector segments according to the zero value situation in the vector segments; a vector inner product circuit, configured to perform a vector inner product operation on the paired vector segments in the retained vector segments to obtain corresponding inner product values; and an accumulation circuit, configured to accumulate the inner product values ​​to generate an element in the i-th row and j-th column in the result matrix.

[0007] In a second aspect, the present disclosure provides a chip comprising the circuit of any one of the embodiments of the first aspect.

[0008] In a third aspect, the present disclosure provides a board comprising the chip of any one of the embodiments of the second aspect.

[0009] In a fourth aspect, the present disclosure provides a method for performing sparse matrix multiplication using the matrix multiplication circuit according to any one of the embodiments of the first aspect.

[0010] Through the matrix multiplication circuit provided above, the method, chip and board for performing sparse matrix multiplication using the matrix multiplication circuit, the disclosed embodiment provides a hardware circuit that supports sparse matrix multiplication, which can greatly reduce the computational complexity by optimizing the multiplication operation process, thereby improving the processing efficiency of the machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an illustrative and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0012] Figure 1 A structural diagram of a board according to an embodiment of the present disclosure is shown;

[0013] Figure 2 A structural diagram showing a combined processing device according to an embodiment of the present disclosure;

[0014] Figure 3 A schematic diagram showing the internal structure of a processor core of a single-core or multi-core computing device according to an embodiment of the present disclosure;

[0015] Figure 4 Show several ways to store sparse matrices;

[0016] Figure 5 A schematic diagram showing the calculation of the inner product of a row vector and a column vector when a sparse matrix exists is shown;

[0017] Figures 6A-6C A schematic diagram illustrating a vector inner product operation according to several embodiments of the present disclosure is shown;

[0018] Figure 7A-7B A schematic diagram illustrating a vector inner product operation according to a further embodiment of the present disclosure; and

[0019] Figure 8 An exemplary structural block diagram of a matrix multiplication circuit according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of this disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this disclosure, not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this disclosure.

[0021] It should be understood that the terms "first," "second," "third," and "fourth," etc., which may appear in the claims, specification, and drawings of this disclosure, are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprising" used in the specification and claims of this disclosure indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0022] It should also be understood that the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the disclosure. As used in this disclosure and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" as used in this disclosure and the claims refers to any and all possible combinations of one or more of the associated listed items, including and including these combinations.

[0023] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context.

[0024] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0025] Figure 1 FIG. 1 is a schematic diagram showing the structure of a board 10 according to an embodiment of the present disclosure. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC), or system-on-chip, integrated with one or more combined processing devices. The combined processing device is an artificial intelligence computing unit that supports various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud-based intelligent applications is the large amount of input data, which places high demands on the platform's storage and computing capabilities. Board 10 of this embodiment is suitable for cloud-based intelligent applications and has extensive off-chip storage, on-chip storage, and powerful computing capabilities.

[0026] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. Calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface formats, such as a PCIe interface.

[0027] Board 10 also includes a memory device 104 for storing data, which includes one or more storage units 105. Memory device 104 is connected to a control device 106 and chip 101 via a bus for data transmission. Control device 106 in board 10 is configured to control the state of chip 101. To this end, in one application scenario, control device 106 may include a microcontroller (MCU).

[0028] Figure 2 FIG. 1 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a storage device 204 .

[0029] The computing device 201 is configured to perform user-specified operations and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0030] Interface device 202 is used to transmit data and control instructions between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to a storage device on-chip of computing device 201. Furthermore, computing device 201 can obtain control instructions from processing device 203 via interface device 202 and write them to a control cache on-chip of computing device 201. Alternatively or optionally, interface device 202 can also read data from the storage device of computing device 201 and transmit it to processing device 203.

[0031] The processing device 203 is a general processing device that performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU) or one or more types of processors in other general and / or special processors, which include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, with respect to the computing device 201 disclosed herein, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0032] The storage device 204 is used to store data to be processed, which may be DRAM or DDR memory, and is typically 16G or larger in size, for storing data of the computing device 201 and / or the processing device 203 .

[0033] Figure 3 The figure shows the internal structure of the processor core when the computing device 201 is a single-core or multi-core device. The computing device 301 is used to process input data for computer vision, speech, natural language, data mining, etc. The computing device 301 includes three modules: a control module 31, a computing module 32, and a storage module 33.

[0034] The control module 31 coordinates and controls the operations of the computing module 32 and the storage module 33 to complete deep learning tasks. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 retrieves instructions from the processing device 203, while the instruction decode unit 312 decodes the retrieved instructions and sends the decoded results as control information to the computing module 32 and the storage module 33.

[0035] The operation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations, supporting complex operations such as vector multiplication, addition, and nonlinear transformations. The matrix operation unit 322 is responsible for the core calculations of the deep learning algorithm, namely matrix multiplication and convolution.

[0036] The storage module 33 is used to store or transfer relevant data and includes a neuron RAM (NRAM) 331, a weight RAM (WRAM) 332, and a direct memory access (DMA) module 333. NRAM 331 stores input neurons, output neurons, and intermediate computational results; WRAM 332 stores the convolution kernels (i.e., weights) of the deep learning network. DMA 333, connected to DRAM 204 via bus 34, is responsible for data transfer between the computing device 301 and DRAM 204.

[0037] The embodiments of the present disclosure provide a matrix multiplication circuit based on the aforementioned hardware environment, which can optimize sparse matrix multiplication operations and greatly reduce the computational complexity.

[0038] In a matrix, if the number of elements with a value of 0 far exceeds the number of non-zero elements, and the distribution of non-zero elements is irregular, the matrix is ​​called a sparse matrix. Simply put, a sparse matrix is ​​a matrix that is mostly 0 and contains only a few non-zero values. The following shows an example of a sparse matrix of size 3×6:

[0039]

[0040] Since most of the elements in a sparse matrix are zero, if it is still stored in the ordinary way, a lot of space will undoubtedly be wasted. At the same time, if the calculation is also performed in the ordinary way, the zero elements will not help the final result, but will increase a lot of invalid calculations.

[0041] For the storage of sparse matrices, in order to achieve the purpose of compression, only non-zero element values ​​(sometimes also called valid element values) are stored, but the positions of non-zero elements must also be retained for easy recovery. Therefore, the storage of sparse matrices not only stores the non-zero element values, but also stores their coordinate positions (row index, column index).

[0042] Figure 4 Shows several ways to store sparse matrices. In other words, Figure 4 Several data structures are provided for storing sparse matrices.

[0043] The COO storage method is called the coordinate format. The COO method uses three arrays to store sparse matrices, which respectively store the row index (row number), column index (column number), and value of the non-zero element coordinates. The length of these three arrays is the number of non-zero elements in the sparse matrix. In theory, the elements in a sparse matrix can be stored in any order, but to facilitate element access, they are stored in the order of left first, right second, and top first, bottom first, that is, row-first storage.

[0044] As shown in the figure, the first element "1" has a row index of 0, a column index of 0, and a value of 1. The second element "2" has a row index of 0, a column index of 1, and a value of 2. Other non-zero elements are stored similarly. As can be seen from the figure, in the COO storage method, there are some duplicate elements in the row index array and the column index array.

[0045] The CSR storage format is called the Compressed Sparse Row Format. The CSR format compresses the row index array in the COO format, leaving the other two arrays unchanged. The three arrays are row pointers, column indices, and values. The lengths of the column index array and the value array remain the number of nonzero elements. The row pointer array stores the offset of the first nonzero element in each row from the first nonzero element in the sparse matrix. Its last element stores the total number of nonzero elements in the sparse matrix. Therefore, the length of the row pointer array is the number of rows in the sparse matrix plus one.

[0046] As shown in the figure, the column index array and value array are the same as the COO method. For the row pointer array, the first element is "0", which means that the first non-zero element "1" in row 0 is offset by 0 from the first non-zero element in the sparse matrix, because it is the first non-zero element itself. The second element "2" means that the first non-zero element "2" in row 1 is offset by 2 from the first non-zero element "1" in the sparse matrix, which is equal to the number of non-zero elements in row 0; the third element "4" means that the first non-zero element "5" in row 2 is offset by 4 from the first non-zero element "1" in the sparse matrix, which is equal to the number of non-zero elements in rows 0 and 1. The fourth element "6" represents the total number of all non-zero elements.

[0047] The CSC storage format, known as the Compressed Sparse Column format, is similar to the CSR format, except that the columns are compressed, leaving the row index and data array unchanged. The column pointer array stores the offset of the first non-zero element in each column from the first non-zero element in the sparse matrix, and its last element stores the total number of non-zero elements in the sparse matrix. Therefore, the length of the column pointer array is the number of columns in the sparse matrix plus one.

[0048] As shown in the figure, the row index array and the value array are the same as the COO method. For the column pointer array, the first element is "0", which means the first non-zero element in column 0. The second element "1" means the first non-zero element "2" in column 1 is offset by 1 from the first non-zero element "1" in the sparse matrix, which is equal to the number of non-zero elements in column 0. The third element "3" means the first non-zero element "4" in column 2 is offset by 3 from the first non-zero element "1" in the sparse matrix, which is equal to the number of non-zero elements in columns 0 and 1. And so on to get other column pointer elements.

[0049] In the embodiments of the present disclosure, when a sparse matrix is ​​provided in any of the above-mentioned compressed storage modes, it can be referred to as a dense form or dense expression of the sparse matrix, or a matrix form consisting of valid data elements (e.g., non-zero values) in the sparse matrix can be referred to as a dense matrix. When a sparse matrix is ​​provided in a dense form, the dense matrix can be desparsed to restore it to the matrix form before the sparse process, such as the example form of the sparse matrix given above, for subsequent processing.

[0050] The following describes the principle of the sparse matrix multiplication operation adopted by the embodiment of the present disclosure.

[0051] According to the definition of matrix multiplication, matrix multiplication can be calculated as follows:

[0052]

[0053] In other words, each data element of the result matrix can be regarded as the vector inner product of the corresponding left-multiplied matrix row vector and the right-multiplied matrix column vector, that is, the vector dot product, which can be simply expressed as the following formula (2):

[0054]

[0055] When at least one matrix in the matrix multiplication operation is a sparse matrix, the above operation method may result in a lot of redundant calculations.

[0056] Figure 5 A schematic diagram showing the calculation of the inner product of a row vector and a column vector when there is a sparse matrix.

[0057] As shown in the figure, the left-multiplied row vector includes 16 elements, but only has 4 non-zero values, whose corresponding indexes are 0, 2, 13, and 15, respectively, indicating that these 4 non-zero values ​​are located in the 0th, 2nd, 13th, and 15th columns of the row, respectively. According to the rules of matrix multiplication, the length of the right-multiplied column vector is the same as the length of the left-multiplied row vector, and also includes 16 elements. In the example in the figure, the right-multiplied column vector has 5 non-zero values, whose corresponding indexes are 1, 3, 7, 12, and 15, respectively, indicating that these 5 non-zero values ​​are located in the 1st, 3rd, 7th, 12th, and 15th rows of the column, respectively.

[0058] In a vector inner product, the values ​​at the corresponding indices of the row and column vectors are multiplied and then summed. As shown in the figure, only the result at index 15 is valid. The remaining results do not contribute to the final result because at least one multiplier is 0. In the example shown, the indices of the data elements in the left and right matrices do not match, resulting in wasted multiplications. Consequently, 15 multiplications are invalid.

[0059] In light of this, embodiments of the present disclosure provide a matrix multiplication circuit that can effectively optimize sparse matrix multiplication operations and avoid wasted computing power. More specifically, embodiments of the present disclosure address vector inner product operations in matrix multiplication steps by pre-processing zero value checks or index matching checks to avoid some invalid multiplication operations, thereby reducing wasted computing power and improving processing efficiency.

[0060] Figure 6A A schematic diagram of a matrix multiplication operation according to one embodiment of the present disclosure is shown. In this embodiment, the row and column vectors of the matrix to be operated on are segmented, and a vector inner product operation is performed on each vector segment. The inner product results of each vector segment are then accumulated to obtain the final result. Furthermore, before performing the vector inner product on the vector segments, an all-zero check can be performed to discard vector segments with all zeros, thereby reducing invalid vector inner product operations.

[0061] In this embodiment, the left-multiplied row vector and the right-multiplied column vector are correspondingly divided into several paired vector segments. As shown in the figure, the left-multiplied row vector includes 16 elements, divided evenly into four segments: L1, L2, L3, and L4. The right-multiplied column vector also includes 16 elements, divided evenly into four segments: R1, R2, R3, and R4. According to the definition of matrix multiplication, row vector segment L1 corresponds to column vector segment R1, and the vector inner product is performed between the two. Row vector segment L2 corresponds to column vector segment R2, and the vector inner product is performed between the two. And so on, the four vector segment inner product results are obtained, and these results are then accumulated to obtain the final result.

[0062] Those skilled in the art will appreciate that various divisions can be performed on the vector, as long as it is ensured that the row vector segments and the column vector segments can correspond or be paired, for example, L1 and R1 constitute a vector segment pair (L1, R1). In one implementation, uneven division can be performed based on the distribution of valid elements of the sparse matrix, that is, multiple row vector segments / column vector segments can have different lengths. In another implementation, the length of each segment can be set according to the sparsity rate of the sparse matrix. For example, when the sparsity rate is high, the segment length can be larger, and when the sparsity rate is low, the segment length can be smaller.

[0063] In the disclosed embodiment, a zero value judgment is performed on the divided vector segments, so that the corresponding vector segments are discarded to avoid using the vector segments to perform vector inner product operations.

[0064] In some embodiments, if any vector segment is all zero, the vector segment and its paired vector segment may be discarded. In other words, for a vector segment pair consisting of a row vector segment and a column vector segment, if any of the vector segments is all zero, the vector segment pair is discarded.

[0065] Figure 6A FIG. 1 shows a schematic diagram of the vector inner product operation according to the above embodiment. The discarded vector segment pairs are shown with oblique lines in the figure. Specifically, in the vector segment pair (L2, R2), since L2 is all zero, the vector segment pair is discarded; in the vector segment pair (L3, R3), both L3 and R3 are all zero, so the vector segment pair is discarded. Accordingly, when performing the vector inner product, the discarded vector segment pair does not participate in the operation, and the vector inner product is only performed on the retained vector segments (L1, R2) and (L4, R4). Compared to Figure 5 The operation method of Figure 6A The calculation method can save a lot of computing power.

[0066] In some embodiments, discarded vector segments may not be stored in the storage circuit, while the retained vector segments may be densely stored in the storage circuit, thereby saving storage space. For example, in the above embodiment, the storage circuit may densely store row vector segments L1 and L4, and column vector segments R1 and R4. Since in this embodiment, vector segments are discarded in pairs, all retained vector segments need to be calculated. Row vector segments and matching column vector segments can be sequentially extracted and sent to the vector inner product circuit for vector inner product calculation.

[0067] Considering some application scenarios, such as in the operation of neural networks, the left multiplication matrix is ​​the input data and the right multiplication matrix is ​​the weight data. The weight data needs to be broadcast or reused. The corresponding segment of the weight data cannot be not loaded because a segment of the input data is all zero. Therefore, the entire vector segment pair cannot be discarded.

[0068] For this scenario, in other embodiments, when a vector segment on one side of the multiplication is all zero, the vector segment can be discarded, but the paired vector segment on the other side of the multiplication is not discarded. The paired vector segment is retained and marked so that in subsequent vector inner product operations, the paired vector segment can be extracted from the retained vector segments to perform vector inner product operations, that is, operations are not performed on this non-discarded vector segment.

[0069] In one embodiment, when a row vector segment in the left multiplication matrix is ​​all zero, the row vector segment is discarded, and the paired column vector segment is retained and marked so that only the paired vector segment is extracted for vector inner product operation in subsequent vector inner product operation.

[0070] Figure 6B A schematic diagram of a vector inner product operation according to the above embodiment is shown. In this embodiment, the corresponding row vector segment is discarded based on the all-zero judgment of the row vector segment of the left multiplication matrix, but the matching column vector segment is not discarded. Specifically, since the row vector segments L2 and L3 of the left multiplication matrix are all zero, these two row vector segments are discarded. Accordingly, the row vector segments L1 and L4 are densely stored in the storage circuit. The paired column vector segments are not discarded but retained, so that all column vector segments R1, R2, R3 and R4 are stored in the storage circuit.

[0071] In order to accurately extract the vector segment pairs that need to be operated when performing the vector inner product operation, various methods can be used for identification so that only paired vector segments are extracted for operation during the vector inner product operation.

[0072] In one implementation, the vector segments paired with the discarded vector segments may be marked, for example, by marking the segment numbers of these vector segments, so that these marked vector segments are skipped during extraction. Figure 6B In the example, the row vector segment L1 and the column vector segment R1 can be first extracted and sent to the vector inner product circuit; then the row vector segment L4 can be extracted in sequence, and R2 and R3 can be skipped according to the mark, and the column vector segment R4 can be directly extracted, and the vector segment pair (L4, R4) can be sent to the vector inner product circuit.

[0073] In another implementation, the column vector segments paired with the reserved row vector segments can be marked. For example, based on the segment number of the row vector segment, the column vector segment with the corresponding segment number is extracted and paired with it and sent to the vector inner product circuit. For example, in Figure 6B In the example, when extracting row vector segment L1, it is determined that the column vector segment paired with it is R1, then the column vector segment R1 is extracted for calculation; when extracting row vector segment L4, it is determined that the column vector segment paired with it is R4, then R2 and R3 are skipped and the column vector segment R4 is directly extracted and sent to the vector inner product circuit.

[0074] Those skilled in the art will appreciate that various identification methods may be used to extract paired vector segments from the retained vector segments for subsequent vector inner product operations, thereby saving computing power.

[0075] In another embodiment, when a column vector segment in the right multiplication matrix is ​​all zero, the column vector segment is discarded, and the paired row vector segment is retained and marked so that only the paired vector segment is extracted for vector inner product operation in subsequent vector inner product operation.

[0076] Figure 6C A schematic diagram of the vector inner product operation according to the above embodiment is shown. In this embodiment, the corresponding column vector segment is discarded based on the all-zero judgment of the column vector segment of the right multiplication matrix, but the row vector segment matching it is not discarded. Specifically, since only R3 of the column vector segment of the right multiplication matrix is ​​all zero, only this column vector segment is discarded. Accordingly, the column vector segments R1, R2 and R4 are densely stored in the storage circuit. The paired row vector segments are not discarded, but retained. Therefore, all row vector segments L1, L2, L3 and L4 are stored in the storage circuit.

[0077] Similarly, various methods can be used for identification so that only paired vector segments are extracted for operation during vector inner product operation, which will not be repeated here.

[0078] from Figures 6A-6C The optimization results show that Figure 6A The method discards the entire vector segment pair, compared to Figure 6B and Figure 6C This method can more effectively save computing power and storage space. Figure 6B and Figure 6C This approach can be applied to situations where one side of the multiplication needs to be reused. All vector segments can be retained for the side that needs to be reused, thereby avoiding repeated loading. The number of discarded vector segments determines the computing power and storage space that can be saved, which depends on the side used for all-zero judgment. For example, Figure 6B The amount of discarding depends on the left multiplication matrix, Figure 6C The amount of discard depends on the right multiplication matrix, so the final savings of the two will be different.

[0079] Further, from Figures 6A-6C The optimization results also show that the all-zero judgment can save some computing power, but there is still a waste of computing power, such as the entire operation of the vector segment pair (L1, R1) and part of the operation in the vector segment pair (L4, R4).

[0080] Alternatively or additionally, in some embodiments of the present disclosure, vector segments may be discarded based on index matching judgment of elements within the vector segments to further save computing power and storage space.

[0081] In some embodiments, if the indices of the non-zero data elements in any vector segment do not match the indices of the non-zero data elements in its paired vector segment, the vector segment and its paired vector segment may be discarded. The indices here indicate the positions of the non-zero data elements within the vector segment. In other words, for a vector segment pair consisting of a row vector segment and a column vector segment, if the indices of their non-zero data elements do not match, the vector segment pair is discarded.

[0082] In some implementations, the above index is a global index, for example, indicating the position of the non-zero data element in the entire matrix, that is, each non-zero data element has a unique index, such as Figures 6A-6B In some other implementations, the above index is a local index, such as the index of the non-zero data element in the vector segment, that is, the index in each vector segment is reused. For example, when the vector segment length is 4, the index in each vector segment can be 0, 1, 2 and 3, as shown in Figure 7A shown.

[0083] Figure 7A FIG. 1 shows a schematic diagram of a vector inner product operation according to the above embodiment. The discarded vector segment pairs are shown with oblique lines in the figure. Figure 6A In the example of , in addition to discarding vector segment pairs where either vector segment is all zero (it can be understood that any vector segment being all zero can be considered a special case of index mismatch), the vector segment pair (L1, R1) is also discarded because the indices of the non-zero data elements in L1 are 0 and 2, while the indices of the non-zero data elements in R1 are 1 and 3. There is no match between the two, so this vector segment pair can be discarded. Accordingly, the storage circuit densely stores the retained vector segments, here, row vector segment L4 and column vector segment R4. Since vector segments are discarded in pairs, the row vector segments and matching column vector segments can be sequentially extracted and transmitted to the vector inner product circuit for vector inner product operation.

[0084] and Figure 6A Compared with the embodiment, Figure 7A This embodiment can further save computing power and storage space. Although there is still some waste of computing power (for example, the multiplication of the first three elements in L4 and R4), compared with the unoptimized solution, it can greatly improve the computing efficiency, so this part of the waste is acceptable.

[0085] and Figure 6B and Figure 6C Similarly, considering that, for example, in the operation of a neural network, the left multiplication matrix is ​​the input data and the right multiplication matrix is ​​the weight data, the weight data needs to be broadcast or reused. The corresponding segment of the weight data cannot be loaded just because a segment of the input data is all zero, so the entire vector segment pair cannot be discarded.

[0086] For this scenario, in other embodiments, when the index of the non-zero data element in the vector segment of one side of the multiplication does not match the index of the non-zero data element in its paired vector segment, the vector segment can be discarded, but the paired vector segment in the other side of the multiplication is not discarded. The paired vector segment is retained and marked so that in subsequent vector inner product operations, the paired vector segment can be extracted from the retained vector segments to perform vector inner product operations, that is, operations are not performed on this non-discarded vector segment.

[0087] Figure 7B A schematic diagram of the vector inner product operation according to the above embodiment is shown. In this embodiment, all vector segments in the right multiplication matrix need to be retained. Based on the index matching judgment of the non-zero data elements in the vector segments, the vector segment pairs (L1, R1), (L2, R2) and (L3, R3) do not have non-zero data elements with index matching, so the row vector segments L1, L2 and L3 in the left multiplication matrix are discarded, but the column vector segments matching them are not discarded. Accordingly, the row vector segment L4 and all the column vector segments R1, R2, R3 and R4 are densely stored in the storage circuit.

[0088] Similarly, various ways of identification can be adopted so that only paired vector segments are extracted for operation during vector inner product operation. For example Figure 7B In the example, only the vector segment pair (L4, R4) is extracted according to the identification and sent to the vector inner product circuit. The specific identification method can be referred to the previous description.

[0089] It can be understood that when it is necessary to retain all vector segments of the left multiplication matrix, the final saving of computing power and storage space is the same as that of retaining the right multiplication matrix. This is because Figure 7B The embodiment discards vector segments based on their index matching, rather than just based on the all-zero condition on one side of the multiplication.

[0090] contrast Figure 7A-7B and Figures 6A-6C The optimization results show that Figure 7A-7B The embodiment can further save computing power and storage space.

[0091] As can be seen from the above description, the matrix multiplication solution provided by the embodiment of the present disclosure can effectively reduce the amount of calculation and save storage space. In the above operation process, only the segmentation circuit and the judgment circuit need to be added, and the existing hardware is slightly modified, which is highly feasible. Since the amount of calculation is directly reduced, the hardware resource overhead is small. In some preferred embodiments (for example Figure 7A-7B In the embodiment where the entire vector segment pair is directly discarded (e.g., Figure 6A and Figure 7AIn the embodiment where only one vector segment is discarded (for example, Figure 6B 、 Figure 6C and Figure 7B ), the scheduling of vector inner products of vector segment pairs can be achieved through simple identification.

[0092] Figure 8 FIG. 8 is a block diagram illustrating an exemplary structure of a matrix multiplication circuit according to an embodiment of the present disclosure. As shown in the figure, the matrix multiplication circuit 800 mainly includes four parts: a segmentation circuit 810 , a control circuit 820 , a vector inner product circuit 830 , and an accumulation circuit 840 .

[0093] The segmentation circuit 810 can be configured to correspondingly divide the i-th row vector of the left multiplication matrix and the j-th column vector of the right multiplication matrix into a plurality of paired vector segments, where i and j are natural numbers. The control circuit 820 can be configured to discard the corresponding vector segments based on the zero value situation in the vector segments. It can be understood that these discarded vector segments are vector segments that will generate invalid operations. The vector inner product circuit 830 is configured to perform a vector inner product operation on the paired vector segments in the retained vector segments to obtain corresponding inner product values. Finally, the accumulation circuit 840 can be configured to accumulate the above inner product values ​​to generate the element in the i-th row and j-th column in the result matrix.

[0094] In some embodiments, the control circuit 820 may be configured to determine which vector segments should be discarded based on a determination of whether a vector segment is all zero. For example, in one implementation, when the control circuit 820 determines that any vector segment is all zero, the vector segment and its paired vector segment may be discarded. In another implementation, when the control circuit 820 determines that any vector segment of either the left multiplication matrix or the right multiplication matrix is ​​all zero, the vector segment may be discarded, but the vector segment paired with the vector segment in the other matrix may be retained. Furthermore, the control circuit 820 also performs an identification so that, based on the identification, the paired vector segment is extracted from the retained vector segments to perform a subsequent vector inner product operation.

[0095] Alternatively or additionally, in some other embodiments, the control circuit 820 may be configured to determine which vector segments should be discarded based on an index matching judgment of the vector segments, where the index is used to indicate the position of the non-zero data element within the vector segment. For example, in one implementation, when the control circuit 820 determines that the index of the non-zero data element in any vector segment does not match the index of the non-zero data element in its paired vector segment, the vector segment and the vector segment paired with it are discarded. In another implementation, when the control circuit 820 determines that the index of the non-zero data element in any vector segment of either the left multiplication matrix or the right multiplication matrix does not match the index of the non-zero data element in its paired vector segment, the vector segment is discarded, but the vector segment paired with the vector segment in the other matrix is ​​retained. Furthermore, the control circuit 820 also performs identification so as to extract the paired vector segments from the retained vector segments to perform the vector inner product operation.

[0096] In some embodiments, identification can be performed in any of the following ways: marking the vector segments paired with the discarded vector segments so that the marked vector segments are skipped during extraction; or marking the vector segments paired with the discarded vector segments in one party based on the retained vector segments in the other party.

[0097] Optionally or additionally, the matrix multiplication circuit 800 may further include a storage circuit 850 for storing data related to the operation. Such data may include, for example, data before, during, and / or after the operation. In some embodiments, the storage circuit 850 may compactly store the retained vector segments, thereby saving storage space.

[0098] In some embodiments, the sparse matrix involved in the matrix multiplication operation may be provided in a dense form, such as CSR, CSC, or COO, which can be referred to as a dense representation of the sparse matrix. In this case, the matrix multiplication circuit 800 may further provide an anti-sparse circuit 860, which is configured to restore the densely represented matrix to be operated to the matrix shape before the sparsification process, so as to provide it to the segmentation circuit for segmentation.

[0099] In some implementations, the anti-sparse circuit 860 can place each data element in the dense expression at the corresponding position of the data shape before the sparsification processing according to the position indicated by the index part in the dense expression and the data shape before the sparsification processing, and fill the remaining positions with zeros, thereby restoring the matrix expression of the sparse matrix.

[0100] Despite Figure 8 In the figure, each circuit is shown as a separate module, but those skilled in the art will appreciate that these circuits can also be implemented as one module or more modules, and the present disclosure is not limited in this respect. For example, the anti-sparse circuit 860, the segmentation circuit 810, and the control circuit 820 can be implemented as one module, such as Figure 3The vector inner product circuit 830 and the accumulation circuit 840 can be implemented as one module, for example, by Figure 3 It is implemented by the operation module in .

[0101] As can be seen from the above description, the embodiments of the present disclosure provide a matrix multiplication circuit that adopts a vector inner product scheme, which can effectively support sparse matrix multiplication. In some embodiments, by segmenting the row vectors and column vectors participating in the matrix multiplication, performing invalid operation judgment based on the vector segments, and discarding the entire vector segments, the waste of computing power can be effectively reduced. In some implementations, the discarded vector segments are determined based on the all-zero condition of the vector segments. Furthermore, in other implementations, the discarded vector segments are determined based on the index matching condition of the vector segments, thereby further reducing invalid calculations. The hardware implementation of the matrix multiplication circuit of the embodiments of the present disclosure is simple, the scheduling complexity of the vector inner product operation is low, and it is easy to implement. In addition, some embodiments of the present disclosure also provide methods, chips and boards for performing sparse matrix multiplication using the matrix multiplication circuit, which include features corresponding to the features described above for the matrix multiplication circuit, which will not be repeated here.

[0102] According to different application scenarios, the electronic devices or devices disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, Internet of Things terminals, mobile terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, automatic driving terminals, vehicles, household appliances, and / or medical equipment. The vehicles include airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging (MRI), ultrasound machines and / or electrocardiographs. The electronic devices or devices disclosed herein may also be applied to the Internet, Internet of Things, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, medical care and other fields. Furthermore, the electronic devices or devices disclosed herein may also be used in cloud, edge, terminal and other application scenarios related to artificial intelligence, big data and / or cloud computing. In one or more embodiments, electronic devices or apparatuses with high computing power according to the disclosed solution can be applied to cloud devices (such as cloud servers), while electronic devices or apparatuses with low power consumption can be applied to terminal devices and / or edge devices (such as smartphones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0103] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will understand that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teachings of the present disclosure, those skilled in the art will understand that some of the steps therein can be performed in other orders or simultaneously. Further, those skilled in the art will understand that the embodiments described in the present disclosure can be regarded as optional embodiments, that is, the actions or modules involved therein are not necessarily necessary for the implementation of one or more schemes of the present disclosure. In addition, depending on the different schemes, the description of some embodiments of the present disclosure also has different emphases. In view of this, those skilled in the art will understand that the parts that are not described in detail in a certain embodiment of the present disclosure may also refer to the relevant descriptions of other embodiments.

[0104] In terms of specific implementation, based on the disclosure and teachings of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented in other ways not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article splits them based on the consideration of logical functions, and there may be other ways of splitting them in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in the units or components can be selectively disabled. With respect to the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0105] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network elements. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the solution described in the embodiments of this disclosure. In addition, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically separately.

[0106] In some other implementation scenarios, the above-mentioned integrated units can also be implemented in the form of hardware, that is, as specific hardware circuits, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which can be, for example, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM and RAM, etc.

[0107] The foregoing may be better understood in light of the following claims:

[0108] Clause 1. A matrix multiplication circuit, wherein at least one of a left-multiplied matrix and a right-multiplied matrix is ​​a sparse matrix, the circuit comprising:

[0109] A segmentation circuit is configured to correspondingly segment the i-th row vector of the left multiplication matrix and the j-th column vector of the right multiplication matrix into a plurality of paired vector segments, where i and j are natural numbers;

[0110] a control circuit configured to discard corresponding vector segments according to a zero value condition in the vector segments;

[0111] a vector inner product circuit configured to perform a vector inner product operation on paired vector segments in the retained vector segments to obtain corresponding inner product values; and

[0112] The accumulation circuit is configured to accumulate the inner product values ​​to generate an element in the i-th row and j-th column in the result matrix.

[0113] Clause 2. The circuit of clause 1, wherein the control circuit is further configured to:

[0114] When any vector segment is all zero, the vector segment and its paired vector segment are discarded.

[0115] Clause 3. The circuit of clause 1 or 2, wherein the control circuit is further configured to:

[0116] When the index of a non-zero data element in any vector segment does not match the index of a non-zero data element in its paired vector segment, wherein the index indicates the position of the non-zero data element within the vector segment, the vector segment and its paired vector segment are discarded.

[0117] Clause 4. The circuit of clause 1, wherein the control circuit is further configured to:

[0118] When any vector segment of one of the left multiplication matrix and the right multiplication matrix is ​​all zero, discard the vector segment and retain the vector segment paired with the vector segment in the other matrix; and

[0119] Identify so that paired vector segments are extracted from the retained vector segments to perform vector inner product operations.

[0120] Clause 5. The circuit of clause 1 or 4, wherein the control circuit is further configured to:

[0121] When the index of a non-zero data element in any vector segment of one of the left multiplication matrix and the right multiplication matrix does not match the index of a non-zero data element in its paired vector segment, discard the vector segment and retain the vector segment paired with the vector segment in the other matrix, wherein the index indicates the position of the non-zero data element in the vector segment; and

[0122] Identify so that paired vector segments are extracted from the retained vector segments to perform vector inner product operations.

[0123] Clause 6. The circuit of clause 4 or 5, wherein the control circuit is configured to identify in any of the following ways:

[0124] Marking the vector segment that is paired with the discarded vector segment so that the marked vector segment is skipped during extraction; or

[0125] Based on the vector segments retained in the one party, the vector segments paired with the vector segments in the other party are marked.

[0126] Clause 7. The circuit according to any one of clauses 4 to 6, wherein the one side is a left-multiplied matrix and the other side is a right-multiplied matrix.

[0127] Clause 8. The circuit according to any one of clauses 1-7, further comprising:

[0128] The storage circuit is configured to densely store the reserved vector segments.

[0129] Clause 9. The circuit of any one of clauses 1-8, wherein the sparse matrix is ​​provided in a dense representation, and the circuit further comprises:

[0130] The anti-sparse circuit is configured to restore the densely expressed matrix to be operated into a matrix expression before the sparse processing to provide it to the segmentation circuit.

[0131] Clause 10. The circuit of clause 9, wherein the anti-sparse circuit is further configured to:

[0132] According to the position indicated by the index part in the dense expression, and in accordance with the data shape before the sparsification process, each data element in the dense expression is placed in the corresponding position of the data shape before the sparsification process, and zeros are filled in the remaining positions of the data shape.

[0133] Clause 11. A circuit according to any one of clauses 9-10, wherein the dense representation is represented by any one of the following methods: a compressed sparse row (CSR) method, a compressed sparse column (CSC) method, and a coordinate COO method.

[0134] Clause 12. A chip comprising the circuit according to any one of clauses 1-11.

[0135] Clause 13. A board comprising the chip according to clause 12.

[0136] Clause 14. A method of performing sparse matrix multiplication using the matrix multiplication circuit according to any one of clauses 1-11.

[0137] The above is a detailed introduction to the embodiments of the present disclosure. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method and core ideas of the present disclosure. At the same time, for those skilled in the art, based on the ideas of the present disclosure, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. A matrix multiplication circuit, wherein at least one of a left-multiplied matrix and a right-multiplied matrix is ​​a sparse matrix, the circuit comprising: a segmentation circuit configured to correspondingly segment the i-th row vector of the left multiplication matrix and the j-th column vector of the right multiplication matrix into a plurality of paired vector segments, where i and j are natural numbers, wherein the vectors are unevenly segmented according to a distribution of valid elements of the sparse matrix, or the length of each vector segment is set according to a sparsity ratio of the sparse matrix; a control circuit configured to discard corresponding vector segments according to a zero value condition in the vector segments; a vector inner product circuit configured to perform a vector inner product operation on paired vector segments in the retained vector segments to obtain corresponding inner product values; as well as The accumulation circuit is configured to accumulate the inner product values ​​to generate an element in the i-th row and j-th column in the result matrix.

2. The circuit of claim 1 , wherein the control circuit is further configured to: When any vector segment is all zero, the vector segment and its paired vector segment are discarded.

3. The circuit according to claim 1 or 2, wherein the control circuit is further configured to: When the index of a non-zero data element in any vector segment does not match the index of a non-zero data element in its paired vector segment, wherein the index indicates the position of the non-zero data element within the vector segment, the vector segment and its paired vector segment are discarded.

4. The circuit of claim 1 , wherein the control circuit is further configured to: When any vector segment of one of the left multiplication matrix and the right multiplication matrix is ​​all zero, discard the vector segment and retain the vector segment paired with the vector segment in the other matrix; and Identify so that paired vector segments are extracted from the retained vector segments to perform vector inner product operations.

5. The circuit according to claim 1 or 4, wherein the control circuit is further configured to: When the index of a non-zero data element in any vector segment of one of the left multiplication matrix and the right multiplication matrix does not match the index of a non-zero data element in its paired vector segment, discard the vector segment and retain the vector segment paired with the vector segment in the other matrix, wherein the index indicates the position of the non-zero data element in the vector segment; and Identify so that paired vector segments are extracted from the retained vector segments to perform vector inner product operations.

6. The circuit according to claim 5, wherein the control circuit is configured to identify in any of the following ways: marking the vector segments that are paired with the discarded vector segments so that the marked vector segments are skipped during extraction; or Based on the vector segments retained in the one party, the vector segments paired with the vector segments in the other party are marked.

7. The circuit according to claim 4, wherein the one side is a left multiplication matrix and the other side is a right multiplication matrix.

8. The circuit of claim 1 , further comprising: The storage circuit is configured to densely store the reserved vector segment.

9. The circuit of claim 1 , wherein the sparse matrix is ​​provided in a dense representation, and the circuit further comprises: The anti-sparse circuit is configured to restore the densely expressed matrix to be operated into a matrix expression before the sparse processing to provide it to the segmentation circuit.

10. The circuit of claim 9, wherein the anti-sparse circuit is further configured to: According to the position indicated by the index part in the dense expression, and in accordance with the data shape before the sparsification process, each data element in the dense expression is placed in the corresponding position of the data shape before the sparsification process, and zeros are filled in the remaining positions of the data shape.

11. The circuit according to any one of claims 9-10, wherein the dense expression is represented by any one of the following methods: compressed sparse row (CSR) method, compressed sparse column (CSC) method, and coordinate COO method.

12. A chip comprising the circuit according to any one of claims 1-11.

13. A board comprising the chip according to claim 12.

14. A method of performing sparse matrix multiplication using the matrix multiplication circuit according to any one of claims 1-11.

Images