Matrix multiplication circuit, method and related products

Abstract

The present disclosure discloses a matrix multiplication circuit, a method for performing sparse matrix multiplication by using the matrix multiplication circuit, and related products. The matrix multiplication circuit can be implemented in a combined processing device including a computing device. The combined processing device can also include an interface device and other processing devices. The computing device interacts with the other processing devices to jointly complete a user-specified computing operation. The combined processing device can also include a storage device connected to the computing device and the other processing devices, respectively, for storing data of the computing device and the other processing devices. The scheme of the present disclosure provides a circuit supporting sparse matrix multiplication, which can simplify processing and improve the processing efficiency of the machine.

Description

Technical Field

[0001] This disclosure generally relates to the field of data processing. More specifically, this disclosure relates to matrix multiplication circuits, methods for performing sparse matrix multiplication using matrix multiplication circuits, chips, and boards. Background Technology

[0002] Sparse matrices account for a certain proportion in data processing. For example, deep learning algorithms, which have developed rapidly in recent years, are computationally intensive and storage-intensive tools. As information processing tasks become increasingly complex and the requirements for the real-time performance and accuracy of algorithms continue to increase, neural networks are often designed to be deeper and deeper, resulting in ever-increasing computational and storage requirements. This makes it difficult to directly apply existing deep learning-based artificial intelligence technologies to mobile phones, satellites, or embedded devices with limited hardware resources.

[0003] Therefore, the compression, acceleration, and optimization of deep neural network models have become particularly important. Sparsity is one such method for lightweighting models. Network parameter sparsity reduces redundant components in larger networks through appropriate methods, thereby reducing the network's computational and storage requirements. This network parameter sparsity will produce a sparse matrix.

[0004] The properties of sparse matrices allow them to significantly reduce computational and storage complexity by leveraging data structure characteristics. Therefore, research and hardware implementation of sparse matrix storage and computation methods can greatly improve the processing performance of related data structures. Existing hardware cannot fully utilize the characteristics of sparse matrices to effectively support their related operations. Summary of the Invention

[0005] In order to at least partially solve one or more of the technical problems mentioned in the background art, the present disclosure provides a matrix multiplication circuit, a method for performing sparse matrix multiplication using the matrix multiplication circuit, a chip, and a board.

[0006] In a first aspect, this disclosure discloses a matrix multiplication circuit, comprising: a vector scalar multiplication circuit configured to perform multiplication operations on each data element in each row of a left-multiplied matrix with the corresponding row vector of a right-multiplied matrix to obtain a plurality of intermediate row vectors, wherein at least one of the left-multiplied matrix and the right-multiplied matrix is ​​a compact matrix composed of effective data elements in a sparse matrix; and a vector sorting and accumulating circuit configured to sort and accumulate the plurality of intermediate row vectors obtained based on data elements in the same row of the left-multiplied matrix to generate a fused row vector as the corresponding row vector in the result matrix.

[0007] In a second aspect, this disclosure provides a chip that includes the circuitry of any of the embodiments of the first aspect.

[0008] In a third aspect, this disclosure provides a board including the chip of any of the embodiments of the second aspect above.

[0009] In a fourth aspect, this disclosure provides a method for performing sparse matrix multiplication using a matrix multiplication circuit according to any embodiment of the first aspect.

[0010] By providing the matrix multiplication circuit, the method for performing sparse matrix multiplication using the matrix multiplication circuit, the chip, and the board as described above, this disclosure provides a hardware circuit that supports sparse matrix multiplication, which can greatly reduce computational complexity by optimizing the multiplication process, thereby improving the processing efficiency of the machine. Attached Figure Description

[0011] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0012] Figure 1 This diagram shows the structure of the board card according to an embodiment of this disclosure;

[0013] Figure 2 This diagram illustrates the structure of the combined processing apparatus according to an embodiment of the present disclosure.

[0014] Figure 3 A schematic diagram illustrating the internal structure of a processor core in a single-core or multi-core computing device according to embodiments of the present disclosure;

[0015] Figure 4 Several storage methods for sparse matrices are shown;

[0016] Figure 5 An exemplary structural block diagram of a matrix multiplication circuit according to an embodiment of this disclosure is shown; and

[0017] Figure 6 An exemplary circuit diagram of a vector sorting and accumulating circuit according to an embodiment of this disclosure is shown. Detailed Implementation

[0018] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0019] It should be understood that the terms "first," "second," "third," and "fourth," etc., that may appear in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0020] It should also be understood that the terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0021] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0022] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

[0023] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0024] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0025] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0026] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a storage device 204.

[0027] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0028] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0029] Processing device 203, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0030] Storage device 204 is used to store data to be processed. It may be DRAM or DDR memory, typically 16G or larger in size, and is used to store data of computing device 201 and / or processing device 203.

[0031] Figure 3 The diagram shows the internal structure of the processor core when the computing device 201 is a single-core or multi-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three main modules: a control module 31, an arithmetic module 32, and a storage module 33.

[0032] The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 fetches instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.

[0033] The computation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 322 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0034] Storage module 33 is used to store or move relevant data, including neuron RAM (NRAM) 331, weight RAM (WRAM) 332, and direct memory access (DMA) module 333. NRAM 331 is used to store input neurons, output neurons, and intermediate results after computation; WRAM 332 is used to store the convolution kernels of the deep learning network, i.e., the weights; DMA 333 is connected to DRAM 204 through bus 34 and is responsible for data transfer between computing device 301 and DRAM 204.

[0035] Based on the aforementioned hardware environment, the embodiments disclosed herein provide a matrix multiplication circuit that can optimize sparse matrix multiplication operations and greatly reduce computational complexity.

[0036] In a matrix, if the number of elements with the value 0 far exceeds the number of non-zero elements, and the distribution of non-zero elements is irregular, then the matrix is ​​called a sparse matrix. Simply put, a sparse matrix is ​​a matrix where the vast majority of elements are 0, containing only a small number of non-zero values. Below is an example of a sparse matrix of size 3×6:

[0037]

[0038] Since sparse matrices are mostly composed of zero elements, storing them in the usual way would undoubtedly waste a lot of space. At the same time, if calculations are performed in the usual way, the zero elements do not help the final result and instead increase a lot of unnecessary calculations.

[0039] To achieve compression, sparse matrix storage only stores the non-zero element values ​​(sometimes called valid element values), but also retains the positions of the non-zero elements for easy recovery. Therefore, sparse matrix storage not only stores the non-zero element values ​​but also their coordinate positions (row index and column index).

[0040] Figure 4 Several storage methods for sparse matrices are shown. Or, in other words, Figure 4 Several data structures are provided for storing sparse matrices.

[0041] The COO storage method, also known as the coordinate format, uses three arrays to store the sparse matrix. These arrays store the row index (row number), column index (column number), and value of the non-zero elements. The length of each array is the number of non-zero elements in the sparse matrix. Theoretically, the elements in the sparse matrix can be stored in any order, but for ease of access, they are stored in left-to-right, top-to-bottom order—that is, in row-major order.

[0042] As shown in the figure, for the first element "1", its row index is 0, its column index is 0, and its value is 1; for the second element "2", its row index is 0, its column index is 1, and its value is 2; other non-zero elements are stored similarly. It can be seen from the figure that in this COO storage method, there are some duplicate elements in the row index array and the column index array.

[0043] CSR storage, also known as Compressed Sparse Row Format, compresses the row index array in the COO format while leaving the other two arrays unchanged. This results in three arrays: row pointers, column indices, and values. The lengths of the column index array and the value array remain the same, counting the number of non-zero elements. The row pointer array stores the offset of the first non-zero element in each row from the first non-zero element in the sparse matrix, and its last element stores the total number of non-zero elements in the sparse matrix. Therefore, the length of the row pointer array is the number of rows in the sparse matrix plus one.

[0044] As shown in the diagram, the column index array and the numeric array are the same as in the COO method. For the row pointer array, the first element is "0", which means that the first non-zero element "1" in row 0 is offset by 0 from the first non-zero element in the sparse matrix, since it is the first non-zero element itself. The second element "2" means that the first non-zero element "2" in row 1 is offset by 2 from the first non-zero element "1" in the sparse matrix, which is equal to the number of non-zero elements in row 0. The third element "4" means that the first non-zero element "5" in row 2 is offset by 4 from the first non-zero element "1" in the sparse matrix, which is equal to the number of non-zero elements in rows 0 and 1. The fourth element "6" represents the total number of all non-zero elements.

[0045] CSC storage, also known as Compressed Sparse Column Format, is similar to CSR, except that columns are compressed while row indices and the data array remain unchanged. The column pointer array stores the offset of the first non-zero element of each column from the first non-zero element of the sparse matrix, and its last element stores the total number of non-zero elements in the sparse matrix. Therefore, the length of the column pointer array is the number of columns in the sparse matrix plus 1.

[0046] As shown in the figure, the row index array and the value array are the same as in the COO method. For the column pointer array, the first element is "0", which represents the first non-zero element in column 0. The second element "1" indicates that the first non-zero element "2" in column 1 is offset by 1 from the first non-zero element "1" in the sparse matrix, which is equal to the number of non-zero elements in column 0; the third element "3" indicates that the first non-zero element "4" in column 2 is offset by 3 from the first non-zero element "1" in the sparse matrix, which is equal to the number of non-zero elements in columns 0 and 1, and so on, to obtain the other column pointer elements.

[0047] In the embodiments disclosed herein, when a sparse matrix is ​​provided in any of the above-described compressed storage methods, it can be referred to as a compact form of the sparse matrix, or a matrix form composed of valid data elements (e.g., non-zero values) in the sparse matrix can be referred to as a compact matrix. When a sparse matrix is ​​provided in a compact form, the compact matrix can be partitioned in the row or column direction for subsequent processing.

[0048] For example, in one embodiment of this disclosure, the matrix can be divided along the row direction according to the storage method of the compact matrix, thereby outputting the divided row vectors and corresponding column indices for processing.

[0049] Still with Figure 4 Taking a matrix as an example, after the compact matrix composed of valid data elements (1,2,3,4,5,6) is row-splittered, three row vectors (1,2), (3,4) and (5,6) are output. Each row vector has a corresponding column index vector (0,1), (1,2) and (3,5), which respectively indicate the column index of each data element in the corresponding row vector.

[0050] The principle of sparse matrix multiplication operation used in the embodiments disclosed herein is described below.

[0051] According to the definition of matrix multiplication, matrix multiplication can be calculated using formula (1):

[0052]

[0053] Each row of the resulting matrix can be viewed as a row vector with a length equal to the number of columns. By extracting the common factors from the row vectors, the resulting matrix can be expressed as formula (2):

[0054]

[0055] As can be seen from formula (2), the i-th row vector of the resulting matrix can be regarded as the sum of multiple row vectors obtained by performing scalar multiplication of each element of the i-th row of the left-multiplied matrix with the corresponding row vector in the right-multiplied matrix. In other words, matrix multiplication can be broken down into two steps: vector scalar multiplication and vector accumulation.

[0056] The above operation method also applies to sparse matrix multiplication. Formulas (3), (4) and (5) show the calculation formulas for at least one sparse matrix in sparse matrix multiplication.

[0057]

[0058]

[0059]

[0060]

[0061] As can be seen from the above formula, when the matrix involved in the multiplication operation is a sparse matrix, only the valid data elements (or non-zero elements) can participate in the first step of vector scalar multiplication, which means that dense computation can be achieved without loss of computational power, and the result of each operation is meaningful. However, in the second step of vector accumulation, column index aligned accumulation, or vector sorting accumulation, is required.

[0062] Based on the above principles, this disclosure provides a matrix multiplication circuit that can effectively optimize sparse matrix multiplication operations and avoid wasting computing power. More specifically, this disclosure provides a hardware implementation scheme for vector sorting and accumulation in the matrix multiplication step, thereby reducing the scheduling complexity and time overhead of vector sorting and accumulation, and effectively improving processing efficiency.

[0063] Figure 5 An exemplary structural block diagram of a matrix multiplication circuit according to an embodiment of this disclosure is shown. As shown, the matrix multiplication circuit 500 mainly includes two parts: a vector scalar multiplication circuit 510 and a vector sorting and accumulating circuit 520.

[0064] The vector-scalar multiplication circuit 510 is configured to perform dot product operations between row vectors and scalars. Specifically, circuit 510 performs multiplication operations on each data element in each row of the left-multiplied matrix with the corresponding row vector of the right-multiplied matrix, thereby obtaining multiple intermediate row vectors. As can be seen from the previous formula, the number of row vectors depends on the number of valid data elements in that row of the left-multiplied matrix, i.e., the number of non-zero elements, and the number of non-all-zero row vectors in the right-multiplied matrix.

[0065] The vector scalar multiplication circuit 510 can be implemented using various existing circuits, such as... Figure 3 The vector operation unit 321 in this disclosure is not limited in this respect.

[0066] The vector sorting and accumulating circuit 520 is configured to perform the sorting and accumulating of the aforementioned intermediate row vectors. Specifically, the circuit 520 sorts and accumulates multiple intermediate row vectors obtained based on the data elements of the same row in the left-multiplied matrix to generate a fused row vector, which serves as the corresponding row vector in the result matrix.

[0067] Furthermore, the matrix multiplication circuit 500 may also include a control circuit 530 for controlling the data elements fed to the vector scalar multiplication circuit 510 and the vector sorting and accumulating circuit 520. As mentioned earlier, each data element in the matrix may have associated row and column indices indicating its position within the matrix. Therefore, the control circuit 530 can control the data elements or vectors fed to the vector scalar multiplication circuit 510 and the vector sorting and accumulating circuit 520 based on these row and / or column indices.

[0068] In some embodiments, the control circuit 530 can control the vector scalar multiplication circuit 510 to perform the following operations: perform multiplication operations on the K data elements in the i-th row of the left multiplication matrix and the corresponding K row vectors in the right multiplication matrix respectively to obtain K intermediate row vectors. In each multiplication operation, the column index of the data element in the left multiplication matrix is ​​equal to the row index of the row vector in the right multiplication matrix. i and K are both natural numbers, and K is the number of data elements in each row of the left multiplication matrix, that is, the number of non-zero elements.

[0069] In some embodiments, the control circuit 530 can control the vector sorting and accumulating circuit 520 to perform the following operations: merging the vector elements in the above K intermediate row vectors into an ordered fused row vector according to their column index order, wherein vector elements with the same column index are accumulated into a fused vector element in the fused row vector, and the fused row vector is used as the row vector of the i-th row of the result matrix of the matrix multiplication.

[0070] Furthermore, the matrix multiplication circuit 500 may also include a preprocessing circuit 540. When the sparse matrix involved in the multiplication operation is provided in a compact form, the preprocessing circuit 540 may be configured to perform row-wise partitioning according to the storage method of the compact matrix, thereby outputting the partitioned row vectors and corresponding column indices for processing.

[0071] Despite Figure 5 The control unit 530 and the preprocessing circuit 540 are shown as two separate modules, but those skilled in the art will understand that these two units can also be implemented as one or more modules, and this disclosure is not limited in this respect.

[0072] Figure 6An exemplary circuit diagram of a vector sorting and accumulating circuit according to an embodiment of this disclosure is shown. Vector sorting and accumulating can be viewed as a data fusion process; therefore, in the following description, "fusion" and "sorting and accumulating" can be understood to have the same meaning based on the context.

[0073] As shown in the figure, in one embodiment, the storage circuit can be exemplarily divided into two parts: a first storage circuit 622 and a second storage circuit 624.

[0074] The first storage circuit 622 can be configured to store K-channel data to be fused and the corresponding K-channel indices, where K>1. The index elements in these K-channel indices indicate the index information of the corresponding data elements in the K-channel data, that is, there is a one-to-one correspondence between the index elements and the data elements.

[0075] In this disclosed embodiment, the K-way data refers to the K intermediate row vectors mentioned above, and the K-way index refers to the column index of each data element in the K intermediate row vectors, wherein the index element in the k-th index corresponds to the column index of each vector element in the k-th intermediate row vector, and k = 1...K.

[0076] Furthermore, the index elements of each of these K-way indices are arranged in a first order, such as ascending order, and the data elements of each of the K-way data are arranged in the order of their corresponding indices. The figure exemplarily illustrates four-way indices and their corresponding four-way data; each data element is identified by the symbol D, and its specific value is not shown. In some embodiments, each index can be stored contiguously, for example, as a vector, so that the index / index vector can be accessed based on the starting address of the index or the starting address of the vector. Correspondingly, each data path can also be stored contiguously, for example, as a vector, so that the data / data vector can be accessed based on the starting address of the data or the starting address of the vector.

[0077] The second storage circuit 624 can be configured to store the fused data output by the vector sorting and accumulating circuit and the corresponding fused index. The fused index elements are arranged in a second order, and the fused data elements are arranged in the order of the fused indexes. As shown in the example, the four data streams to be fused become one fused data stream, and the corresponding four indices also become one fused index. The fused index elements are arranged in ascending order, and index elements of the same size are removed. The corresponding fused data elements are arranged in the order of the fused indexes, and data elements with the same index are accumulated to form the fused data element.

[0078] In some embodiments, the vector sorting and accumulating circuit 600 may include a sorting circuit 632 and an accumulating circuit 636 to collaboratively implement the sorting and accumulating function. Specifically, the sorting circuit 632 sorts the K-way indices according to the size of the index elements and outputs them in order to the accumulating circuit 636. The accumulating circuit 636, when receiving identical index elements from the sorting circuit, accumulates the data elements corresponding to the identical index elements and removes duplicate index elements.

[0079] In some embodiments, the sorting circuit 632 may include a comparison circuit 631 and a buffer circuit 633. The comparison circuit 631 performs a comparison function, comparing the sizes of index elements in multiple indices to be merged and submitting the comparison results to the control circuit 610 for sorting. The control circuit 610 determines the insertion position of the index element in the buffer circuit 633 based on the comparison results. The buffer circuit 633 is used to cache the compared index elements and their corresponding data elements, caching them in order of index element size.

[0080] Specifically, the comparison circuit 631 can be configured to compare the index elements in the index to be merged with the index elements not yet output in the buffer circuit 633, and output the comparison result to the control circuit 610. The buffer circuit 633 can be configured to, according to the control of the control circuit 610, orderly store the information of the compared index elements and their corresponding data elements, and orderly output the information of the compared index elements and their corresponding data elements.

[0081] In some embodiments, buffer circuit 633 can be configured to cache K index elements, which are sorted by size. Those skilled in the art will understand that the buffer circuit can also be configured to cache more index elements, and the embodiments disclosed herein are not limited in this respect. Depending on the sorting method in buffer circuit 633 and the desired output sorting method, such as ascending or descending, the first or last index element in the current sequence can be output in a specified order each time. For example, in the example in the figure, buffer circuit 633 caches index elements from left to right in descending order, outputting the rightmost index element each time, which is the smallest index element in the current sequence, such as "7".

[0082] In these embodiments, the comparison circuit 631 may include a K-1 comparator configured to compare the index element to be fused with the index elements not yet output in the buffer circuit 633, that is, to compare with the K-1 index elements remaining after the first or last index element of the current sequence is output, generate a comparison result and output it to the control circuit 610.

[0083] For example, for four data streams to be merged, a three-way comparator is shown in the figure, which compares the specified index element (9 in this case) received from the first storage circuit 622 with the three index elements that are not currently output from the buffer circuit 633. The three index elements on the left in the figure are 100, 10 and 9.

[0084] In some embodiments, the comparison result of the comparator can be represented using a bitmap. For example, if the index element to be merged (e.g., 9) is greater than or equal to the index element in the buffer circuit, the comparator can output "1"; otherwise, it outputs "0"; and vice versa. In the example in the figure, the comparison result of the index element to be merged (9) with the respective index elements (100, 10, and 9) in the buffer circuit is "001", which is output to the control circuit 610.

[0085] The control circuit 610 can be configured to determine the insertion position of the index element to be merged in the current sequence of the buffer circuit 633 based on the received comparison result. Specifically, the control circuit 610 can be further configured to determine the insertion position based on the change position of the bits in the bitmap. In the example shown in the figure, the comparison result is "001", indicating that the index element to be merged is less than the first and second index elements from the left in the buffer circuit, and greater than or equal to the third index element from the left. Therefore, the insertion position is between the second and third index elements, that is, between "10" and "9".

[0086] In some embodiments, buffer circuit 633 may be configured to insert the index element to be merged at the insertion position according to the instruction of control circuit 610. In the example in the figure, the sequence after the index element is inserted in buffer circuit 633 becomes "100,10,9,9".

[0087] To ensure that the data corresponding to the index can be retrieved during the merge sort accumulation process, in some embodiments, the buffer circuit 633 can be further configured to: orderly store the compared index elements and their corresponding data elements according to the value order of the index elements. As shown in the figure, the buffer circuit 633 caches not only the index elements but also the information of their corresponding data elements. Therefore, after each comparison of the index elements to determine the insertion position, the information of the data element corresponding to that index element can also be inserted into the buffer circuit. Those skilled in the art will understand that the information of the data element can be the data element itself, such as D32, D23, etc., as exemplarily shown in the figure; the information of the data element can also be an address pointing to the data element, and the embodiments disclosed herein are not limited in this respect.

[0088] Next, the buffer circuit 633 can output the rightmost index element "9". At this time, the control circuit 610 can be further configured to determine the memory access information of the next index element to be merged based on the index element output from the buffer circuit. Specifically, the control circuit retrieves the next index element to be merged from which index in the K-way index the output index element belongs, and sends it to the comparison circuit 631 for comparison.

[0089] Furthermore, during ordered output, the buffer circuit 633 can be configured to output the compared index elements in order of their values ​​(e.g., from smallest to largest) as fusion indices, and simultaneously output their corresponding data elements as fusion data. The output data is provided to the accumulation circuit 636 for further processing.

[0090] For clarity, the diagram also shows the index sequence cached in buffer circuit 633 as the sorting progresses. As shown, initially, the first index element of each of the K-way indices is stored in buffer circuit 633 in descending order. In some implementations, these four index elements can be retrieved, sorted, and stored in the buffer circuit all at once. In other implementations, the data in the buffer circuit can be initialized to negative numbers, and the first index element of each indices can be retrieved sequentially (e.g., from indices 1 to 4), compared with the data in the buffer circuit, and placed in the appropriate position. In this example, the first index element of all four indices is 0, so they can be arranged according to the index number based on the order of retrieval; for example, the "0" of indices 1 is placed on the far right, the "0" of indices 2 is placed in the second position from the right, and so on.

[0091] Next, the rightmost "0" belonging to the first channel in the buffer circuit is output. Based on which channel this output index element belongs to, the next index element to be merged is retrieved from that channel, namely the second index element "2" of the first channel. "2" is sent to the comparison circuit and compared with the remaining three "0"s in the buffer circuit. The comparison result is "111", which is greater than all three existing "0"s in the buffer circuit. Therefore, "2" is inserted at the end of the sequence, and the sequence in the buffer circuit becomes "2,0,0,0".

[0092] Next, the rightmost "0" belonging to the second path in the buffer circuit is output. Therefore, the second element "3" of the second path is taken out and compared with the remaining "2,0,0" in the buffer circuit. The comparison result is "111", so "3" is inserted at the end of the sequence. At this time, the sequence in the buffer circuit becomes "3,2,0,0".

[0093] Next, the rightmost "0" belonging to the third path in the buffer circuit is output. Therefore, the second element of the third path, "100", is taken out and compared with the remaining "2,0,0" in the buffer circuit. The comparison result is "111", so "100" is inserted at the end of the sequence. At this time, the sequence in the buffer circuit becomes "100,3,2,0".

[0094] Next, the rightmost "0" belonging to the 4th channel of the output buffer circuit is taken out and the second element "2" of the 4th channel is compared with the remaining "100,3,2" in the buffer circuit. The comparison result is "001", so "2" is inserted after the rightmost first element of the sequence. At this time, the sequence in the buffer circuit becomes "100,3,2,2".

[0095] Similarly, the index elements in the K-way index can be compared one by one, sorted according to size, and inserted into the appropriate positions in the buffer circuit before being output by the buffer circuit. For example, the smallest index element output by the buffer circuit each time can be output sequentially to the accumulator circuit 636. Those skilled in the art will understand that if the buffer circuit has sufficient space, the merged and sorted elements can also be output uniformly after the sorting is completed.

[0096] As can be seen from the merged sorted index elements, when there are index elements of the same size, the sorting circuit 632 still retains these index elements of the same size and does not perform deduplication. Instead, it provides them to the accumulation circuit 636 for processing.

[0097] In some embodiments, the accumulator circuit 636 may include a comparator 637, a buffer 635, and an adder 639.

[0098] Comparator 637 can be configured to compare the index elements output in order from sorting circuit 632 with the previous merged index element and output the comparison result. The comparison result can be "1" indicating that they are the same, "0" indicating that they are different, or vice versa.

[0099] Buffer 635 can be configured to control the output index element based on the comparison result of comparator 637. In some embodiments, buffer 635 may output the current index element as a new fused index element only when the comparison results indicate that they are different. In other words, when the comparison results indicate that they are the same, buffer 635 does not output the current index element, that is, it discards the index element that is the same as the previous fused index element. As shown, there are no duplicate fused index elements in the fused index of the second storage circuit 624.

[0100] Adder 639 can be configured to control the accumulation of data elements based on the comparison result of comparator 637. Specifically, when the comparison results indicate that they are different, the data element corresponding to the current index element is directly output as the new fused data element; when the comparison results indicate that they are the same, the data element corresponding to the current index element is accumulated onto the fused data element corresponding to the previous fused index element.

[0101] By comparing the current index element with the previous fused index element each time the output of the sorting circuit 632 is received, the same index elements can be processed differently, thereby realizing the functions of index deduplication and data accumulation.

[0102] As described above, this disclosure provides a matrix multiplication circuit that effectively supports sparse matrix multiplication. Further, this disclosure provides a vector scalar multiplication circuit that performs operations on non-zero elements in a sparse matrix; all operations are valid, thus eliminating computational waste. Even further, this disclosure provides a vector sorting and accumulation circuit that supports sorting and accumulating vectors in the aforementioned operations, thereby reducing the computational power loss due to unaligned vectorization. The matrix multiplication circuit of this disclosure significantly reduces computational complexity by optimizing the multiplication process, thereby improving machine processing efficiency. In addition, some embodiments of this disclosure also provide methods, chips, and boards for performing sparse matrix multiplication using this matrix multiplication circuit, which include features corresponding to those described above for the matrix multiplication circuit, and will not be repeated here.

[0103] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0104] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0105] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0106] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0107] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0108] The foregoing can be better understood in accordance with the following terms:

[0109] Clause 1. A matrix multiplication circuit, comprising:

[0110] A vector scalar multiplication circuit is configured to perform multiplication operations on each data element in each row of the left-multiplied matrix with the corresponding row vector of the right-multiplied matrix, resulting in multiple intermediate row vectors, wherein at least one of the left-multiplied matrix and the right-multiplied matrix is ​​a compact matrix composed of the effective data elements of the sparse matrix; and

[0111] A vector sorting and accumulating circuit is configured to sort and accumulate multiple intermediate row vectors obtained based on data elements in the same row of a left-multiplied matrix, so as to generate a fused row vector as the corresponding row vector in the result matrix.

[0112] Clause 2. The circuit according to Clause 1, wherein each data element has an associated row index and column index for indicating the position information of the data element in the unsparse matrix, the circuit further comprising:

[0113] A control circuit configured to control the delivery of data elements or vectors to the vector scalar multiplication circuit and the vector sorting and accumulating circuit based on the row index and / or column index.

[0114] Clause 3. The circuit described in Clause 2, wherein:

[0115] The vector scalar multiplication circuit is further configured to, under the control of the control circuit, perform multiplication operations on the K data elements in the i-th row of the left-multiplied matrix and the corresponding K row vectors in the right-multiplied matrix, respectively, to obtain K intermediate row vectors. In each multiplication operation, the column index of the data element in the left-multiplied matrix is ​​equal to the row index of the row vector in the right-multiplied matrix, where i and K are both natural numbers, and K is the number of data elements in each row of the left-multiplied matrix.

[0116] The vector sorting and accumulating circuit is further configured to, under the control of the control circuit, merge the vector elements in the K intermediate row vectors into an ordered fused row vector according to their column index order, wherein vector elements with the same column index are accumulated to form a fused vector element in the fused row vector, and the fused row vector serves as the row vector of the i-th row of the result matrix.

[0117] Clause 4. The circuit according to Clause 3, wherein the vector sorting and accumulating circuit comprises a sorting circuit and an accumulating circuit, wherein

[0118] The sorting circuit is configured to sort the K-way indices corresponding to the K intermediate row vectors by size and output them in an ordered manner to the accumulation circuit, wherein the index element in the k-th index corresponds to the column index of each vector element in the k-th intermediate row vector, and k = 1…K; and

[0119] The accumulation circuit is configured to accumulate the vector elements corresponding to the same index elements when the same index elements are received from the sorting circuit, and remove duplicate index elements.

[0120] Clause 5. The circuit according to Clause 4, wherein the sorting circuit includes a comparison circuit and a buffer circuit, wherein:

[0121] The comparison circuit is configured to compare the index elements to be sorted in the K-way index with the index elements not yet output in the buffer circuit, and output the comparison result to the control circuit; and

[0122] The buffer circuit is configured to, under the control of the control circuit, orderly store the information of the compared index elements and their corresponding vector elements, and orderly output the information of the compared index elements and their corresponding vector elements.

[0123] Clause 6. The circuit according to Clause 5, wherein the comparison circuit comprises:

[0124] A K-1-way comparator is configured to compare the index elements to be sorted in the K-way index with the K-1 index elements of the current sequence in the buffer circuit, generate comparison results, and output them to the control circuit.

[0125] Clause 7. The circuit according to Clause 6, wherein the control circuit is configured to determine, based on the comparison result, the insertion position of the index element to be sorted in the current sequence of the buffer circuit.

[0126] Clause 8. The circuit according to Clause 7, wherein the comparison result is represented using a bitmap, and the control circuit is further configured to: determine the insertion position based on the positional changes of bits in the bitmap.

[0127] Clause 9. The circuit according to any one of Clauses 7-8, wherein the buffer circuit is configured to insert, according to the instruction of the control circuit, the information of the index element to be sorted and its corresponding data element at the insertion position.

[0128] Clause 10. The circuit according to any one of Clauses 5-9, wherein the buffer circuit is further configured to output information of the first or last index element in the current sequence and its corresponding vector element in a specified order.

[0129] Clause 11. The circuit according to Clause 10, wherein the control circuit is further configured to: determine memory access information for the next index element to be sorted based on the index element output in the buffer circuit.

[0130] Clause 12. A circuit according to any one of Clauses 4-11, wherein the accumulator circuit comprises a comparator, a buffer, and an adder, wherein:

[0131] The comparator is configured to compare the index element output from the sorting circuit with the previous fused index element and output the comparison result.

[0132] The buffer is configured to output the index element as a new fusion index element only when the comparison results indicate a difference; and

[0133] The adder is configured to directly output the vector element corresponding to the index element as a new fusion vector element when the comparison results indicate that they are different, and to accumulate the vector element corresponding to the index element to the fusion vector element corresponding to the previous fusion index element when the comparison results indicate that they are the same.

[0134] Clause 13. A circuit according to any one of Clauses 1-12, wherein the dense matrix is ​​stored in any of the following manner: Compacted Sparse Row (CSR) storage, Compacted Sparse Column (CSC) storage, and Coordinate-on-O (COO) storage, and the circuit further comprises:

[0135] The preprocessing circuit is configured to perform row partitioning on the compact matrix according to the storage method of the compact matrix, and output the partitioned row vectors and corresponding column indices for processing.

[0136] Clause 14. A chip comprising the circuitry described in any one of Clauses 1-13.

[0137] Clause 15. A board card including the chip described in Clause 14.

[0138] Clause 16. A method for performing sparse matrix multiplication using a matrix multiplication circuit according to any one of Clauses 1-13.

[0139] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A matrix multiplication circuit, comprising: A vector scalar multiplication circuit is configured to perform multiplication operations on each data element in each row of the left-multiplied matrix with the corresponding row vector of the right-multiplied matrix to obtain multiple intermediate row vectors. The left-multiplied matrix and the right-multiplied matrix are both compact matrices composed of effective data elements from sparse matrices. The corresponding row vector of the right-multiplied matrix is ​​obtained by dividing the compact matrix in the row direction, and the data elements in the corresponding row vector of the right-multiplied matrix are all effective data elements. as well as A vector sorting and accumulating circuit is configured to sort and accumulate multiple intermediate row vectors obtained based on data elements in the same row of a left-multiplied matrix, so as to generate a fused row vector as the corresponding row vector in the result matrix.

2. The circuit of claim 1, wherein each data element has an associated row index and column index for indicating the position information of the data element in the unsparsed matrix, the circuit further comprising: A control circuit configured to control the delivery of data elements or vectors to the vector scalar multiplication circuit and the vector sorting and accumulating circuit based on the row index and / or column index.

3. The circuit according to claim 2, wherein: The vector scalar multiplication circuit is further configured to, under the control of the control circuit, perform multiplication operations on the K data elements in the i-th row of the left-multiplied matrix and the corresponding K row vectors in the right-multiplied matrix, respectively, to obtain K intermediate row vectors. In each multiplication operation, the column index of the data element in the left-multiplied matrix is ​​equal to the row index of the row vector in the right-multiplied matrix, where i and K are both natural numbers, and K is the number of data elements in each row of the left-multiplied matrix. The vector sorting and accumulating circuit is further configured to, under the control of the control circuit, merge the vector elements in the K intermediate row vectors into an ordered fused row vector according to their column index order, wherein vector elements with the same column index are accumulated to form a fused vector element in the fused row vector, and the fused row vector serves as the row vector of the i-th row of the result matrix.

4. The circuit according to claim 3, wherein the vector sorting and accumulating circuit comprises a sorting circuit and an accumulating circuit, wherein... The sorting circuit is configured to sort the K-way indices corresponding to the K intermediate row vectors by size and output them in an ordered manner to the accumulation circuit, wherein the index element in the k-th indices corresponds to the column index of each vector element in the k-th intermediate row vector, k=1…K; and The accumulation circuit is configured to accumulate the vector elements corresponding to the same index elements when the same index elements are received from the sorting circuit, and remove duplicate index elements.

5. The circuit according to claim 4, wherein the sorting circuit comprises a comparison circuit and a buffer circuit, wherein: The comparison circuit is configured to compare the index elements to be sorted in the K-way index with the index elements not yet output in the buffer circuit, and output the comparison result to the control circuit; and The buffer circuit is configured to, under the control of the control circuit, orderly store the information of the compared index elements and their corresponding vector elements, and orderly output the information of the compared index elements and their corresponding vector elements.

6. The circuit of claim 5, wherein the comparison circuit comprises: A K-1-way comparator is configured to compare the index elements to be sorted in the K-way index with the K-1 index elements of the current sequence in the buffer circuit, generate comparison results, and output them to the control circuit.

7. The circuit of claim 6, wherein the control circuit is configured to determine, based on the comparison result, the insertion position of the index element to be sorted in the current sequence of the buffer circuit.

8. The circuit of claim 7, wherein the comparison result is represented using a bitmap, and the control circuit is further configured to: determine the insertion position based on the change position of the bits in the bitmap.

9. The circuit according to any one of claims 7-8, wherein the buffer circuit is configured to insert information of the index element to be sorted and its corresponding data element at the insertion position according to the instruction of the control circuit.

10. The circuit according to any one of claims 5-8, wherein the buffer circuit is further configured to output information of the first or last index element in the current sequence and its corresponding vector element in a specified order.

11. The circuit of claim 10, wherein the control circuit is further configured to: determine memory access information of the next index element to be sorted based on the index element output in the buffer circuit.

12. The circuit according to any one of claims 4-8, wherein the accumulation circuit comprises a comparator, a buffer, and an adder, wherein: The comparator is configured to compare the index element output from the sorting circuit with the previous fused index element and output the comparison result. The buffer is configured to output the index element as a new fusion index element only when the comparison results indicate a difference; and The adder is configured to directly output the vector element corresponding to the index element as a new fusion vector element when the comparison results indicate that they are different, and to accumulate the vector element corresponding to the index element to the fusion vector element corresponding to the previous fusion index element when the comparison results indicate that they are the same.

13. The circuit according to any one of claims 1-8, wherein the dense matrix is ​​stored in any of the following manner: compressed sparse row (CSR) storage, compressed sparse column (CSC) storage, and coordinate (COO) storage, and the circuit further comprises: The preprocessing circuit is configured to perform row partitioning on the compact matrix according to the storage method of the compact matrix, and output the partitioned row vectors and corresponding column indices for processing.

14. A chip comprising the circuit according to any one of claims 1-13.

15. A circuit board comprising the chip according to claim 14.

16. A method for performing sparse matrix multiplication using a matrix multiplication circuit according to any one of claims 1-13.

Images